An observability system based on business context augmentation

Abstract

The application relates to an observable system based on service context enhancement, which is built based on an observable architecture and comprises a data acquisition layer, a data processing layer, a data storage layer and a data display layer. The data acquisition layer is used for acquiring metadata of service requirements and technical requirements, and generating technical monitoring data under a service context. The data processing layer is used for associating enhanced data including external service data and software life cycle data with the technical monitoring data under the service context to generate derivative data. The data processing layer is also used for performing semantic analysis on the derivative data to generate pragmatic data, wherein the pragmatic data includes service level operation steps in a natural language form and service optimization suggestions. The data storage layer is used for storing the derivative data and the pragmatic data. The data display layer is used for displaying the derivative data and the pragmatic data. Compared with the prior art, the application can realize deep integration and comprehensive observability of technical data and service processes, and reduce communication barriers between technical teams and service teams.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of observability, and particularly relates to a business context enhanced observability system. BACKGROUND

[0002] Large enterprises need the cooperation of technical teams and business teams to ensure the rapid and stable development of the enterprise. Among them, the technical team is responsible for maintaining the IT system of the enterprise to ensure its smooth operation, and the business team focuses on market trends, customer needs and business strategies, and uses the IT system of the enterprise to handle business problems. However, it is difficult for the technical team to associate a large amount of technical data with the actual business impact, and it is difficult to respond to the needs of the business team in a timely manner; it is also difficult for the business team to understand the complex data provided by the technical team, and it is difficult to use technical support to handle abnormal business in a timely manner. The main reasons are as follows:

[0003] First, the current technical team widely uses observability technology to ensure the stable operation of the IT system. Through logs, performance indicators and tracking information, the technical team can understand the running state of the system in real time to ensure the efficient operation of the system. However, the system based on traditional observability technology mainly focuses on the technical level, lacks business context, and the data such as business process, user experience and market response are separated from IT data, which leads to difficulties in cooperation between departments when dealing with risks or problems.

[0004] Second, the business monitoring system of the business team is often independent of the technical monitoring system, and the information island phenomenon is serious. The business department can only rely on result data such as transaction results to judge the health status of the business, and lacks process data support. This separation increases the difficulty of discovering potential risks in a timely manner, making the troubleshooting and optimization of business problems complex and inefficient.

[0005] In summary, it is necessary to design an observability system to realize the deep integration and comprehensive observability of technical data and business processes, reduce the communication barriers between technical teams and business teams, and improve the observability, auditing ability and abnormal response level of business and technology. SUMMARY

[0006] The purpose of the present application is to overcome the defects of the prior art and provide a business context enhanced observability system, which realizes the deep integration and comprehensive observability of technical data and business processes, reduces the communication barriers between technical teams and business teams, and improves the observation, auditing and abnormal disposal ability of industry-technology integration.

[0007] The purpose of the present application can be achieved by the following technical solutions:

[0008] The application provides a kind of observability system based on service context enhancement, based on the construction of observability architecture, including data acquisition layer, data processing layer, data storage layer and data display layer, the data acquisition layer is used to collect the metadata of service demand and technical demand, to generate technical monitoring data under the service context accordingly;The data processing layer is used to associate enhanced data with technical monitoring data under the service context, generate structure-unified derivative data, the enhanced data includes external business data and software life cycle data;The data processing layer is also used to perform semantic analysis on the derivative data, to generate pragmatic data, the pragmatic data includes business level operation step and business optimization suggestion in natural language form;The data storage layer is used to store the derivative data and the pragmatic data;The data display layer is used to display the derivative data and the pragmatic data in visual form.

[0009] Further, the data acquisition layer includes business context management module, business context injection module and dynamic configuration management module, the business context management module is used to obtain the metadata of service demand and technical demand, and to establish business data dictionary accordingly, the business data dictionary includes Span information, Span attribute and log information under the service context;The business context injection module integrates IDE plug-in, to inject the information in the business data dictionary into the tracking unit and log of the observability architecture in a structured manner, to obtain technical monitoring data under the service context;The dynamic configuration management module is used to dynamically adjust the data acquisition strategy of the business context management module through double-mode strategy.

[0010] Further, the establishment process of the business data dictionary is as follows:

[0011] The business process name is extracted from the business demand as the name of tracking unit Span, to establish Span information table, the Span information table includes business process name, business meaning and business action;

[0012] Based on business demand and technical demand, Span attribute table is established, the Span attribute table includes field information of business entity, life cycle state and association identifier;

[0013] The association action between entities is extracted from business demand and technical demand, to establish log information table, the log information table includes log number, association action name and log template, the log template includes associated entity, behavior type, value change and additional information.

[0014] Further, the data acquisition strategy of the business context management module is loaded and applied through CI / CD component, the process of dynamically adjusting the data acquisition strategy of the business context management module by the double-mode strategy of the dynamic configuration management module is as follows:

[0015] The sampling rate, filtering rules and data buffer size of data collection are dynamically adjusted through the configuration interface provided by the observability SDK.

[0016] Whether data needs to be collected is determined by an observability function defined based on context information and pre-defined rules, the context information including current business process, user type and system load.

[0017] Further, the data processing layer includes a derived data offline processing module and an intelligent pragmatic analysis processing module, the derived data offline processing module being configured to obtain enhanced data and associate it with technical monitoring data in the pre-processed business context to generate derived data, and the intelligent pragmatic analysis processing module being configured to perform semantic analysis on the derived data through a large language model to generate pragmatic data.

[0018] Further, the specific process of generating derived data is as follows:

[0019] Business entity IDs in the technical monitoring data in the pre-processed business context are extracted, based on which external business data is obtained, and the pre-processed technical monitoring data in the business context is associated with the external business data through primary key matching, foreign key reference or consistency hash-based methods;

[0020] Key fields in the pre-processed technical monitoring data in the business context are extracted, and software lifecycle data including commit hash, configuration file version and build environment information is obtained based on the publish package ID in the key fields; the pre-processed technical monitoring data in the business context is associated with the software lifecycle data through hash matching, mapping table or indexing mechanism;

[0021] According to the hierarchical relationship, association relationship and query requirements of data, the associated data is uniformly modeled, and a large data processing framework and a distributed search engine are used to generate indexes, and finally derived data is obtained.

[0022] Further, the data storage layer includes a derived data storage module and a pragmatic data storage module, which are respectively configured to receive and store the derived data and the pragmatic data;

[0023] The derived data storage module first determines the target storage backend of the derived data according to the routing rules, and then further determines the transmission protocol and interface according to the target storage backend, and adopts parallel transmission mode for transmission;

[0024] The pragmatic data storage module first collects interaction data of pragmatic data and associates it with corresponding metadata to establish pragmatic data records; and then designs data storage structure to store the pragmatic data records.

[0025] Further, the derived data and the pragmatic data are compressed and selectively encrypted before being transmitted; during transmission, data integrity and data legality are checked; if transmission fails, an error log is recorded and automatic retry is performed.

[0026] Further, if the large language model generating a certain piece of pragmatic data is upgraded, the upgraded large language model is used to generate upgraded pragmatic data based on the interaction data of the pragmatic data, the upgraded pragmatic data is compared with the original pragmatic data, and optimization suggestions for the large language model are generated.

[0027] Further, the data display layer includes a derived data display module and a pragmatic data display module, which are used to display the derived data and the pragmatic data, respectively, and the pragmatic data display module displays the pragmatic data in natural language form through prompting engineering.

[0028] Compared with the prior art, the present application has the following beneficial effects:

[0029] 1. The present application proposes an observability system based on business context enhancement, which is built based on an observability architecture, including a data collection layer, a data processing layer, a data storage layer and a data display layer, wherein the data collection layer is used to collect metadata of business requirements and technical requirements, and generate technical monitoring data under the business context, so that the technical team and the business team can share this set of data, greatly reducing communication barriers and improving the speed of problem troubleshooting and business optimization; the data processing layer is used to deeply associate external business data and software life cycle data with technical monitoring data under the business context, generate derived data with unified structure, comprehensiveness and cross-dimension, and provide strong support for subsequent big data analysis, business insight and prediction, greatly improving the analysis, auditing and early warning capabilities of the system in cross-domain scenarios; the data processing layer is also used to perform semantic analysis on the derived data and generate pragmatic data, which includes business level operation steps and business optimization suggestions in natural language form, which can improve the accuracy and accuracy of analysis, and enable the business team to more intuitively understand complex business processes and system performance, and improve the efficiency of business decision-making and optimization; the data display layer is used to display derived data and pragmatic data in a visual form, which can intuitively observe complex technical monitoring data, realize seamless communication between business and technical personnel, and make business decision-making more intelligent and efficient.

[0030] 2、The application establishes a business data dictionary according to the metadata of business requirements and technical requirements, the business data dictionary includes Span information, Span attributes and log information under the business context, the business process name, attribute information and inter-entity associated actions and other information can be deeply integrated into the technical monitoring system, the deep fusion of the business context and the technical monitoring and the standardized management of the metadata information are realized, and fine-grained analysis under a complex business scenario is supported.

[0031] 3、In the application, the data collection strategy of the business context management module is loaded and applied through the CI / CD component, the data collection strategy of the business context management module is dynamically adjusted through a dual-mode strategy, specifically, the sampling rate, filtering rules and data buffer size of data collection are dynamically adjusted through a configuration interface provided by the observability SDK; whether data needs to be collected is judged through an observability function defined based on context information and pre-defined rules, the context information includes the current business process, user type and system load; the above design can maximize the reduction of unnecessary data while ensuring key data collection, optimize system performance, and the strategy update automatically takes effect with the CI / CD process, which can reduce configuration errors and improve system stability.

[0032] 4、The application generates derived data through the following steps: extracting business entity IDs in the technical monitoring data under the pre-processed business context, obtaining external business data accordingly, associating the technical monitoring data under the pre-processed business context with the external business data through primary key matching, foreign key reference or a consistent hash-based manner; extracting key fields in the technical monitoring data under the pre-processed business context, obtaining software life cycle data based on the release package ID in the key fields, the software life cycle data includes submission hash, configuration file version and build environment information; associating the technical monitoring data under the pre-processed business context with the software life cycle data through hash matching, mapping table or index mechanism; finally, according to the hierarchical relationship, association relationship and query requirement of the data, the associated data is uniformly modeled, and a big data processing framework and a distributed search engine are used to generate an index; the above design can realize the deep association and enrichment of observability data, business data and software life cycle data, improve the abnormal detection efficiency and prediction ability of the system, and is suitable for complex new business scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A structural diagram of an observability system based on business context enhancement. DETAILED DESCRIPTION

[0034] The application will be described in detail below with reference to the drawings and specific embodiments. The embodiments are implemented on the premise of the technical solutions of the application, and detailed implementation modes and specific operation processes are given, but the protection scope of the application is not limited to the following embodiments.

[0035] Embodiments:

[0036] In traditional observability tools, technical data and business data are usually separated, making it difficult for technical personnel to quickly understand the direct impact of system failure on business. The embodiment introduces business context into the observability system, providing an observability system enhanced based on business context, which closely integrates technical data and business processes. The system is built based on an observability architecture (such as OpenTelemetry), without affecting the original performance monitoring, error tracking, and index display functions of the observability architecture. It can continue to use logs, indexes, and tracking functions, as well as existing tools such as Prometheus, Jaeger, Grafana, etc. As shown in the figure, the system includes a data collection layer, a data processing layer, a data storage layer, and a data display layer. The improvement is that business processes, entity information, and operation context are injected into technical monitoring data, so that observability data can not only show system running status, but also closely integrate with business processes to support real-time monitoring and optimization of business. The structure and function of each layer will be described in detail below. Figure 1

[0037] I. Data collection layer

[0038] The data collection layer includes a business context management module, a business context injection module, and a dynamic configuration management module. The business context management module is used to obtain metadata of business requirements and technical requirements, and to establish a business data dictionary accordingly. The business data dictionary includes Span information, Span attributes, and log information under the business context. The business context injection module integrates an IDE plug-in to inject information in the business data dictionary into the tracking unit and logs of the observability architecture in a structured manner, obtaining technical monitoring data under the business context. The dynamic configuration management module is used to dynamically adjust the data collection strategy of the business context management module through a dual-mode strategy. The specific description of each module is as follows:

[0039] (1) Business context management module

[0040] The context management module establishes a unified business data dictionary by standardizing the management of metadata of business requirements and technical requirements. The specific process of establishing the business data dictionary is as follows:

[0041] S111, extract and manage the business process name as the tracking unit name.

[0042] ​The parsing algorithm is used to extract the business process name (e.g., "contract creation", "product classification update", etc.) from the metadata of business requirements as the name of the tracking unit Span, which is recorded in the Span information table. The core data of the Span information table includes the business process name (the name of the Span, corresponding to the name of the business process or process step), the list of related attributes (the list of attribute names and business roles related to the Span), and the description (a detailed description of the Span, explaining its business meaning and role). This step can make each Span have clear business semantics, facilitating the quick alignment of system status between technical and business teams.

[0043] S112, define and manage the declaration of Span attribute metadata.

[0044] Based on the metadata of business requirements and technical requirements, a Span attribute table is established, and the core data of the Span attribute table includes attribute name (the name of the Span attribute, which identifies a specific business or technical indicator) and attribute description (a detailed description of the attribute, explaining its meaning and purpose). Through the Span attribute table, the definition of each attribute can be standardized to ensure data consistency. These attributes may include field information of business entities, lifecycle status, association identifier, etc.

[0045] S113, manage and record the association action categories and templates between entities as log content.

[0046] From the metadata of business requirements and technical requirements, the association action between entities is extracted, and a log information table is established. The core data of the log information table includes log number (unique identifier of log record), association action name (e.g., "product classification update", "order status change", etc.), and log template (defines the format and content of the log, including associated entities, behavior type, value change, additional information, etc.). The log information table can provide more granular business observability, helping technical and business teams quickly locate problems, audit business processes, strengthen monitoring and analysis of business operations, and improve risk control capabilities.

[0047] The business context management module deeply integrates the business process name, attribute information (including business entities, lifecycle, and other domain metadata), and inter-entity association action into the monitoring system, achieving deep integration of business context and technical monitoring, which can enhance business observability, improve problem positioning efficiency, and promote business and technical collaboration. By standardizing the management of metadata information and establishing a complete business data dictionary, fine-grained analysis in complex business scenarios can be supported, which can improve system maintainability, enhance system scalability, and improve data utilization.

[0048] (2) Business context injection module

[0049] The context injection module is responsible for injecting business process information, entity information and associated actions into the trace unit (Span) and log of the observability system in a structured manner. One important function of this module is to provide an IDE plug-in to assist developers in referencing the standards of the context management module during coding to conveniently complete the injection of context information. In this way, the close integration of business requirements and technical observability systems is ensured, forming a digital twin of the system runtime. The data processing process inside this module is as follows:

[0050] S121, name the Span based on the business requirement description.

[0051] When naming each Span, use the business process or sub-process name from the Span information table in the context management module. The specific operation is as follows: the developer manually calls the API of the observability SDK in the code to create and name the Span. When coding, refer to the standards provided by the context management module to name the Span name, ensuring that it meets the standard naming of the business process. The business context injection module provides an IDE plug-in that automatically reads the Span information table of the context management module. In the IDE, the appropriate Span name can be easily selected through a drop-down list, improving the development experience and reducing manual input errors.

[0052] Through the above steps, technical and business personnel can intuitively understand the specific business process being processed by the system when debugging and analyzing, promoting collaboration between business and technical teams.

[0053] S122, record entity classification, entity information and cross-domain association information in the Span attribute.

[0054] The developer injects the key information of the business entity and the relevant data of the cross-domain association into the attributes of the Span in the form of key-value pairs when coding. The specific operation is as follows: use the API of the observability SDK to manually add Span attributes in the code. Refer to the Span attribute table of the context management module to ensure that the attribute name meets the standards. The IDE plug-in of the business context injection module automatically reads the Span attribute table and provides a drop-down list during coding to facilitate developers to select attribute names and corresponding variable values. The plug-in can also prompt the attributes that need to be filled in to ensure the completeness and accuracy of the attributes. For cross-domain association information such as release package version, hash number, etc., inject global association information in the root Span or related Span of the Trace.

[0055] By recording the key information of business entities and cross-domain associations, the observability data is enriched. The system not only captures the key context information in the business process, but also performs correlation analysis across multiple business domains at the data processing layer, providing strong support for in-depth business analysis, problem troubleshooting, and cross-system fault tracking.

[0056] S123, record the inter-entity association action into a log (Log).

[0057] Developers record the association actions between different entities in the business process in the code, generating detailed log information. Specifically, use the API of the observability SDK to manually create log records. Refer to the log information table of the context management module to ensure that the format and content of the log meet the standards. The IDE plug-in of the context injection module automatically reads the log information table, provides a drop-down list, and developers can select the appropriate log template and fill in the necessary variable values. The plug-in can automatically generate log code snippets, further simplifying the development process.

[0058] The IDE plug-in is used to assist developers in conveniently injecting business context information during coding. The plug-in works in the developer's coding environment and interacts with the metadata of the context management module to provide automated support for developers, such as automatic completion, input prompts, and code generation. The main purpose of the plug-in is to automatically complete method names and parameters based on the context during development, provide relevant attribute and method suggestions during coding, and automatically generate code snippets such as chain calls, thereby seamlessly integrating the standard metadata in the context management module into the code, ensuring the accuracy and consistency of business context information.

[0059] The input of the IDE plug-in includes the metadata of the context management module (including the Span information table, the Span attribute table, and the log information table) and the coding environment information of the developer (including the currently edited source code file and location, project configuration, and dependency). The plug-in mainly implements the following functions: ① integration with the context management module: obtaining the latest metadata information from the context management module at startup or regularly, ensuring that the developer uses the latest standard, supporting both manual refresh and automatic synchronization modes, and caching the obtained metadata locally to reduce dependence on remote services and improve response speed; ② user interface interaction: automatic completion and drop-down list, i.e., when the developer calls the relevant methods of the observability SDK in the code, the plug-in automatically provides a drop-down list of available Span names, attributes, and log templates; input prompt and verification, i.e., the plug-in provides input prompts when the developer inputs parameters, including the meaning of the parameters, the expected data type, and examples, identifies mandatory and optional items, and prompts for missing mandatory items when necessary; code snippet generation, i.e., the plug-in can automatically generate the corresponding code snippet based on the selected Span name, attribute, and log template, and insert it into the current cursor position, and the generated code follows the coding specifications and styles of the project, reducing the amount of manual coding; it can automatically generate observable code snippets such as chain calls; ③ integration with the observability SDK: API mapping, i.e., the plug-in is pre-configured with API mapping of the observability SDK, knows how to call methods to create Spans, add attributes, and record logs, supports multiple programming languages and frameworks, and adapts to different development environments; context management, i.e., the plug-in assists in handling the context transfer of Spans, ensures that the created Spans are correctly associated with parent Spans, provides auxiliary tools to help developers manage context objects, and avoids context loss or association errors; ④ configuration and customization: developers can configure synchronization frequency, display options, and shortcut keys through the settings interface of the plug-in, support enabling or disabling specific functions such as automatic code generation and input verification, and the plug-in allows configuration of project coding specifications, log formats, and naming rules to ensure that the generated code meets project requirements.

[0060] The output of the IDE plug-in includes generated code snippets, prompt information during development, and metadata update logs. The code snippets include Span creation code, Span attribute setting code, and log recording code, the prompt information includes errors and warnings, as well as optimization suggestions, and the metadata update logs include synchronization logs and logs recorded by the plug-in to synchronize metadata with the context management module, facilitating developers to view the update status of metadata.

[0061] Through the above design, the IDE plugin can act as a bridge between the business context management module and developers, effectively connecting the metadata of the business context management module with the coding practices of developers. The plugin's auto-completion and code generation functions significantly improve development efficiency, allowing developers to focus on implementing business logic; input suggestions and validation functions help developers promptly identify and correct errors, ensuring code quality; and unified naming and formatting standards facilitate subsequent data processing, analysis, and presentation, ensuring that the injection of business context information into the code conforms to the standards of the context management module, maintaining data consistency and accuracy, and promoting internal team collaboration.

[0062] Example of use:

[0063] ① Create a Span: Developers create a new Span in the code. After entering startSpan, the plugin automatically pops up a drop-down list showing the available business process names. For example, after selecting order creation, the plugin generates the code: Spanspan = tracer.startSpan("order creation");

[0064] ② Add Span attributes: After creating a Span, type setAttribute. The plugin will suggest available attribute names, such as Order ID or User ID. For example, if you select Order ID, the plugin will automatically fill in the code and prompt you to fill in the corresponding variable value: span.setAttribute("Order ID", orderId);

[0065] ③ Logging: In business logic, it is necessary to log a related action. Input `logEvent`, and the plugin provides a choice of log templates. For example, if you select the "Order Status Update" template, the plugin will generate the following code: `Span.addEvent("Order Status Update", attributes);`

[0066] The plugin prompts for the required attributes, such as old state and new state. After the developer fills in the information, the logging is completed.

[0067] (3) Dynamic Configuration Management Module

[0068] The amount of observable data is enormous, and data acquisition and storage costs are high. To optimize system performance and reduce costs, it is necessary to dynamically adjust the data acquisition strategy. This module implements dynamic adjustment of the data acquisition strategy based on the following two methods:

[0069] 1) Effective via the observability SDK.

[0070] The observability SDK provides a configuration interface that allows for dynamic adjustment of the data acquisition sampling rate, filtering rules, and data buffer size. This method directly impacts the original configuration of the observability system, enabling global strategy adjustments.

[0071] 2) Through customized observability functions

[0072] Before data collection, determine whether data needs to be collected through an observability function defined based on context information (including current business processes, user types, and system load) and predefined rules.

[0073] The above means can meet diverse business needs and adapt to different scenarios. In critical business scenarios, it ensures the collection of sufficient fine-grained data to support problem troubleshooting and performance analysis. At the same time, it maximizes the reduction of unnecessary data collection and storage, optimizing system performance.

[0074] Integrate the update of data collection strategy with the continuous integration and continuous delivery (CI / CD) process to ensure that the strategy changes take effect when the application is redeployed. This can quickly respond to business changes, shorten the cycle of strategy adjustment, reduce configuration errors, and improve system stability.

[0075] II. Data processing layer

[0076] The data processing layer includes a derived data offline processing module and an intelligent pragmatic analysis processing module. The derived data offline processing module is used to obtain enhanced data (including external business data and software lifecycle data) and associate it with preprocessed technical monitoring data in the business context to generate derived data. The intelligent pragmatic analysis processing module is used to perform semantic analysis on the derived data through a large language model to generate pragmatic data, which includes business-level operation steps and business optimization suggestions in natural language form. The specific descriptions of each module are as follows:

[0077] (1) Derived data offline processing module

[0078] This module enriches the observability data by deeply associating technical data with business data, software lifecycle data, etc., to generate cross-dimensional derived data, supporting intelligent analysis and prediction in complex business scenarios. The process of processing data by this module is as follows:

[0079] S211, Data collection and preprocessing

[0080] Obtain external business system data (including business data from CRM, ERP, supply chain management systems, etc.), software lifecycle data (including commit information in the code repository, release package ID from the CI / CD system, version information from the configuration management system, etc.), and technical monitoring data in the business context. Clean, format, and standardize the collected data to ensure data quality and consistency.

[0081] S212, Data enrichment

[0082] Correlation of observability data and software lifecycle data: First, extract the key fields (including release package ID, Trace ID, Span ID) in the pre-processed technical monitoring data under the business context; then, call the API of the CI / CD system based on the release package ID to obtain the software lifecycle data (including commit hash, configuration file version and build environment information); secondly, associate the pre-processed technical monitoring data under the business context with the software lifecycle data through hash matching, mapping table or indexing mechanism. The following is an example: when a Trace's root Span contains a release package ID, the corresponding commit hash and related build information can be obtained from the CI / CD system through this ID.

[0083] Correlation of observability data and external business system data: First, extract the business entity ID (such as user ID, order ID, product ID, etc.) in the pre-processed technical monitoring data under the business context; then, obtain external business data through API or data warehouse; secondly, associate the pre-processed technical monitoring data under the business context with the external business data through primary key matching, foreign key reference or consistency hash-based methods. The following is an example: through the user ID, obtain the user's detailed information such as user level, industry, historical purchase records, etc. from the CRM system.

[0084] S213, data index generation and storage

[0085] First, consider the hierarchical relationship, correlation and query requirements of the data, design a unified data model to adapt to multi-dimensional data structure, including technical indicators, business attributes and lifecycle information. Secondly, use big data processing framework (such as Hadoop, Spark) and distributed search engine (such as Elasticsearch) to generate index, index the enriched data, obtain structured derivative data (including Span's hierarchical relationship, time-consuming information, business attributes, etc.), support real-time query and complex data aggregation analysis. Indexing methods include inverted index (inverted index on key fields such as user ID, order ID, TraceID, error code, etc., supporting fast query), sharding index (sharding data by time, business module or other dimensions, distributed storage and processing, improving concurrent query capability) and data compression (using data compression techniques such as Parquet format, columnar storage, optimizing storage space and query performance), all of which can build efficient big data index. Then, store the index and data in a high-performance storage system to support real-time query and complex analysis.

[0086] Through the derived data offline processing module, the original observability data not only records the technical status of the system, but also is enriched by business and life cycle data to form more comprehensive and cross-dimensional derived data. The module describes the input, output and processing process of data enrichment in detail, reaching the implementable standard. It provides strong support for subsequent big data analysis, business insight and prediction, greatly enhancing the intelligent analysis and risk warning capability of the system.

[0087] The following are three use examples:

[0088] Example 1: Error positioning

[0089] After the system detects an error and finds the relevant Span through the Trace ID, the publishing package ID in the Span is extracted and associated with the CI / CD system to obtain the corresponding commit hash. Through the commit hash, the specific code change, developer and code review information are located. Through the above processing, technical personnel can quickly locate the problem source and speed up problem troubleshooting.

[0090] Example 2: User behavior analysis

[0091] If the business team wants to analyze the use experience of VIP users, first, extract the user ID from the observability data and associate it with the CRM system to obtain the user level. Then, in the enriched data set, query the operation track, response time, error situation, etc. of VIP users. Through the above processing, the business team can optimize services for VIP users and improve satisfaction.

[0092] Example 3: Risk warning

[0093] If the system needs to detect possible fraudulent behavior, first, associate the user's historical behavior data through the user ID to analyze the user's behavior pattern in multiple business processes; then, use the enriched multi-dimensional data to apply machine learning algorithms to identify abnormal behavior.

[0094] (2) Intelligent pragmatic analysis and processing module

[0095] The intelligent pragmatic analysis and processing module performs semantic analysis and processing on the collected original observability data, business context information, and structured Span and log information, generates natural language explanations, analysis reports and business optimization suggestions for business personnel, and helps business decision-making and problem positioning. The data processing process inside the module is as follows:

[0096] S221, generate LLM input rich in semantics

[0097] The structured derivative data (containing hierarchical relationships of Span, time-consuming information, business attributes, etc.) after preprocessing (including data parsing, hierarchical organization, data cleaning, and data structuring, etc.) is pushed to the intelligent pragmatic data analysis module through data interfaces or message queues, and is converted into an input format understandable by LLM, such as text description or specific input templates, using human-readable forms to organize data, including necessary context information.

[0098] Example: Construct a description that embodies the hierarchical information of Span:

[0099] A customer with user ID 12345 completes a payment process with a total time of 10.45 seconds. The specific operations include:

[0100] ① Order creation (3.12 seconds)

[0101] - Verify inventory (1.00 seconds): Check if the product inventory is sufficient.

[0102] - Save order (2.12 seconds): Save order information to the database.

[0103] ② Payment confirmation (5.89 seconds)

[0104] - Third-party payment interface call (5.00 seconds): Interact with the payment platform to handle payment.

[0105] - Update order status (0.89 seconds): Update the order status to paid.

[0106] ③ Customer notification (1.44 seconds)

[0107] - Send SMS notification (0.50 seconds): Send a payment success SMS to the customer's mobile phone.

[0108] - Update notification record (0.94 seconds): Record the notification sending situation.

[0109] This description clearly embodies the hierarchical structure of Span, showing the time-consuming and description of each operation step and its sub-step, facilitating in-depth analysis of each link of complex business processes, discovering potential problems and optimization points.

[0110] S222, call large language model (LLM) for semantic analysis and content generation

[0111] First, use a pre-trained large language model (such as GPT-4) to input rich semantic text and set appropriate prompts to guide the LLM to analyze and generate; second, design prompts to let the LLM understand the input content, analyze user behavior, system performance, potential problems, etc.; third, according to the analysis results, generate natural language explanations, analysis reports or business optimization suggestions. Not only can it improve the accuracy and precision of analysis, but also enable business personnel to more intuitively understand complex business processes and system performance, improving the efficiency of business decision-making and optimization. For the example mentioned in S221, the corresponding LLM-generated output is as follows:

[0112] User 12345 spent a total of 10.45 seconds in the payment process. Here is the detailed analysis:

[0113] ① Order creation (3.12 seconds)

[0114] - Verify inventory (1.00 seconds): Time consumption is normal.

[0115] - Save order (2.12 seconds): Time consumption is slightly high, may need to optimize database write performance.

[0116] ② Payment confirmation (5.89 seconds)

[0117] - Third-party payment interface call (5.00 seconds): Time consumption is longer, may be due to slow response from third-party payment platform, suggest communicating with payment platform for optimization.

[0118] - Update order status (0.89 seconds): Time consumption is normal.

[0119] ③ Customer notification (1.44 seconds)

[0120] - Send SMS notification (0.50 seconds): Time consumption is normal.

[0121] - Update notification record (0.94 seconds): Time consumption is normal.

[0122] Suggestions:

[0123] - Optimize the order saving process and check database performance.

[0124] - Contact the third-party payment platform to understand the response of the payment interface and seek optimization solutions.

[0125] - Overall, the payment process time consumption is mainly concentrated in the payment confirmation stage, which needs to be focused on.

[0126] Through the above processing, the intelligent pragmatic analysis processing module makes full use of the hierarchical information of Span, generates input rich in semantics, and provides a large language model for in-depth semantic analysis and content generation. This method not only improves the accuracy and accuracy of the analysis, but also enables business personnel to more intuitively understand complex business processes and system performance, improving the efficiency of business decision-making and optimization.

[0127] III. Data storage layer

[0128] The data storage layer includes a derived data storage module and a pragmatic data storage module, which are used to receive and store derived data and pragmatic data, respectively. The specific description of each module is as follows:

[0129] (1) Derived data storage module

[0130] In order to support diversified business needs and application scenarios, it is necessary to efficiently store derived data to different data backends for various systems and application programs. The data processing process in the derived data storage module is as follows:

[0131] S311, data formatting and standardization

[0132] Unified data format: Ensure that derived data conforms to the format and structure required by each target storage backend, so that derived data can be seamlessly shared and retrieved in different storage systems. Although the standardized JSON format is used, different storage systems may have specific field requirements or limitations, and appropriate format conversion or field mapping is required.

[0133] Field mapping and data cleaning: Map and clean the fields in the data to ensure the accuracy and integrity of the data. For example, unify the date format and filter out null or invalid data.

[0134] S312, data distribution

[0135] According to the routing rules, determine the target storage backend of the derived data, meet the multi-dimensional analysis needs in different business scenarios. For example, for real-time monitoring data, export to Splunk to meet real-time query and monitoring needs; for full-text search needs, export to Elasticsearch / OpenSearch to support full-text search and complex queries; for OLAP analysis needs, export to Doris to support multi-dimensional analysis and reporting.

[0136] S313, data transmission

[0137] According to the target storage backend, further determine the transmission protocol and interface (such as HTTP, TCP, dedicated client, etc.), use efficient parallel transmission mechanism for transmission. Before transmission, compress the data to reduce bandwidth occupation, and encrypt the sensitive data to ensure data security. During transmission, check the data (such as consistency check and transaction control), ensure that the data is not tampered or damaged, and then ensure the consistency of the data of each storage backend. When data transmission fails, automatically retry to ensure the reliability of data transmission, and record the transmission error in detail for subsequent analysis and processing.

[0138] S314, data storage

[0139] If the storage backend is Splunk, use the API or data receiver provided by Splunk to send data to Splunk, which may need to adjust the data format to meet the indexing and search requirements of Splunk. If the storage backend is Elasticsearch / OpenSearch, use the Bulk API of Elasticsearch to realize batch import and indexing of data, define appropriate index structure and mapping (Mapping), and optimize query performance. If the storage backend is Doris: use the Stream Load or Broker Load function of Doris to import data into Doris, configure the table structure of Doris, define the type and partition strategy of the column, and meet the analysis requirements to provide reliable data support for business decision-making.

[0140] (2) Pragmatic data storage module

[0141] This module is used for the systematic storage of all original data, requests (Prompt), return results and related metadata generated during the intelligent pragmatic analysis process. By comprehensively recording the interactive information and analysis process of large language models (LLM), it can better support the upgrading of large models, model improvement and testing based on retrieval enhancement (RAG), etc. This not only helps to continuously optimize the performance of the model, but also provides a solid data foundation for historical analysis, reanalysis and advanced comprehensive analysis, promoting the accuracy of business decision-making.

[0142] The input data of the pragmatic data storage module includes LLM interaction raw data and related metadata. The LLM interaction raw data includes: ① Prompt: the text request sent by the user or system to the large language model, including business context, task instructions, and related data; ② Response: the reply of the large language model to the request, containing the generated analysis report, suggestion, or answer. The related metadata includes: ① Model version information: the version number or identifier of the large language model used; ② Request timestamp: the date and time of the interaction; ③ Requester identity: the user ID, role, or system module name that initiated the request; ④ Conversation ID: the identifier associated with the same conversation if there are multiple rounds of conversation; ⑤ Business scenario: the associated business process name, task type, or operation category; ⑥ Associated entity information: the identification and related information of business entities (such as user ID, order ID). The following is an example input:

[0143] · Prompt:

[0144] Task: Please analyze the data of the following user payment process, identify possible performance bottlenecks, and provide optimization suggestions.

[0145] Data:

[0146] The customer with user ID 12345 completed a payment process, with a total time of 10.45 seconds. The specific operations include:

[0147] ① Order creation (3.12 seconds)

[0148] - Verify inventory (1.00 second): Check if the product inventory is sufficient.

[0149] - Save order (2.12 seconds): Save order information to the database.

[0150] ② Payment confirmation (5.89 seconds)

[0151] - Third-party payment interface call (5.00 seconds): Interact with the payment platform to handle payment.

[0152] - Update order status (0.89 seconds): Update the order status to paid.

[0153] ③ Customer notification (1.44 seconds)

[0154] - Send SMS notification (0.50 seconds): Send a payment success SMS to the customer's mobile phone.

[0155] - Update notification record (0.94 seconds): Record the notification sending situation.

[0156] Please generate a detailed analysis report based on the above data, pointing out the steps with longer time consumption, and providing corresponding optimization suggestions.

[0157] Response:

[0158] Analysis Report:

[0159] User 12345 spent a total of 10.45 seconds in the payment process. Here is the detailed analysis:

[0160] 1. Order creation (3.12 seconds)

[0161] - Inventory verification (1.00 second): Normal time consumption.

[0162] - Order saving (2.12 seconds): Slightly high time consumption, suggest optimizing database write performance.

[0163] 2. Payment confirmation (5.89 seconds)

[0164] - Third-party payment interface call (5.00 seconds): Longer time consumption, possible slow response from payment platform, suggest negotiating with payment platform for optimization.

[0165] - Update order status (0.89 seconds): Normal time consumption.

[0166] 3. Customer notification (1.44 seconds)

[0167] - Send SMS notification (0.50 seconds): Normal time consumption.

[0168] - Update notification record (0.94 seconds): Normal time consumption.

[0169] Overall Suggestions:

[0170] - Optimize the order saving process and check database performance.

[0171] - Communicate with the third-party payment platform to optimize the payment interface response time.

[0172] Related Metadata:

[0173] Model version: GPT-4

[0174] Request timestamp: 2023-10-15 14:32:10

[0175] Requester identity: User ID 67890 (Business Analyst)

[0176] Session ID: session_abc123

[0177] Business scenario: Payment process performance analysis

[0178] Associated entity information: User ID 12345, Order ID O123456

[0179] The data processing process in the pragmatic data storage module is as follows:

[0180] S321, data collection and archiving

[0181] Capture and save each interaction with the large language model, including the request (Prompt) and the return result (Response), associate relevant metadata with the corresponding interaction data, and form a complete record.

[0182] S322, data storage structure design

[0183] Use a database suitable for high-concurrency writing and querying, such as a NoSQL database (MongoDB, Cassandra) or a relational database (MySQL, PostgreSQL). The data model design includes the design of the main table and the index,

[0184] Main table: LLM_Interaction

[0185] Fields:

[0186] interaction_id (primary key)

[0187] prompt_text (request content)

[0188] response_text (return result)

[0189] model_version (model version)

[0190] timestamp (request timestamp)

[0191] requester_id (requester identity)

[0192] session_id (session ID)

[0193] business_context (business scenario)

[0194] entity_info (associated entity information, JSON format)

[0195] Index design: Establish indexes on commonly used query fields, such as timestamp, requester_id, model_version, and business_context.

[0196] Perform integrity checks on data to ensure that both requests and returned results are successfully saved, avoiding data loss. Perform desensitization or encryption on sensitive information to comply with data security and privacy protection requirements. Ensure that data storage complies with relevant laws and regulations, such as GDPR, Network Security Law, etc.

[0197] When the system's large language model is upgraded (e.g., from GPT-4 to GPT-O1), reanalysis is required to evaluate the performance improvement of the new model in business analysis. The processing steps are as follows: First, extract all interaction records from the pragmatic data storage module in the past month, including Prompt and related metadata; then, use GPT-O1 to regenerate the return results for these Prompts; next, store the new return results together with the corresponding Prompt, metadata, and mark the model version as GPT-O1; finally, use the difference analysis tool to summarize the performance improvement of the new model in different business scenarios, pointing out specific improvement points and areas that need to be optimized.

[0198] In intelligent pragmatic analysis, combined with the enterprise internal knowledge base, the RAG technology is used to improve the accuracy of the model's answers. For RAG-based model improvement, in each interaction, record the used external knowledge base content, retrieved reference materials, and RAG-related metadata; when the knowledge base is updated or the RAG strategy is adjusted, use the stored data to re-interact, evaluate the effect of the new strategy, compare the answer quality of the model under the old and new RAG strategies, adjust the knowledge base content or RAG strategy, and further improve the model performance and answer accuracy.

[0199] This module realizes the comprehensive storage of all original data, requests, return results, and business context in the LLM interaction process, and uses the stored historical data to reanalyze and compare results after the upgrade of the large language model or the enhancement of RAG. By completely saving the original data and metadata, it can trace back each analysis process, guarantee the integrity and auditability of the data, and support the continuous optimization of the model.

[0200] IV. Data Display Layer

[0201] The data display layer includes the derived data display module and the pragmatic data display module, which are used to display derived data and pragmatic data respectively. The specific descriptions of each module are as follows:

[0202] (1) Derived Data Display Module

[0203] The derived data presentation module is the UI frontend for derived data in the system, responsible for presenting the enriched and processed derived data to users in an intuitive and user-friendly manner. This module provides high-dimensional data visualization interfaces, integrating business data and technical data, helping users to deeply analyze system performance, business processes, and potential optimization opportunities, supporting business decision-making and problem positioning.

[0204] Data from the derived data storage module, including all structured business and technical data, is transmitted to the derived data presentation module through standardized interfaces, ensuring that the latest business and technical monitoring results are always displayed. The specific explanation is as follows:

[0205] ① Multi-dimensional data presentation:

[0206] This module integrates technical and business data and presents it to users in multiple dimensions. For example, technical data (such as response time, system load) and business data (such as order status, user information) will be displayed on the same interface in the form of charts, tables, or other visualizations. Users can select different data dimensions for filtering and combination, helping them quickly locate business or technical problems.

[0207] Visualization tools: The module supports a variety of visualization tools, such as line charts, pie charts, bar charts, heat maps, etc., helping users intuitively understand data trends and relationships.

[0208] Example: For performance monitoring of a business operation, users can view the relationship between the processing time of each order and business key indicators such as order amount, user type, etc.

[0209] ② Interactive data analysis:

[0210] Users can perform real-time data filtering, sorting, and drill-down analysis through the module. Interactive functions allow users to go from a macro perspective to specific business operations. For example, users can gradually refine from the overall business process monitoring view to the performance data of specific business entities or user operation behavior.

[0211] Real-time data update: The module integrates real-time data update functions, allowing users to view the latest monitoring data without refreshing the page. This ensures that users can respond quickly and make business adjustments based on the latest system status.

[0212] ③ Customized presentation:

[0213] The module allows users to customize data presentation templates according to business needs. Users can select specific data fields of interest, visualization styles, and presentation priorities to ensure that users at different levels (such as technical personnel, business analysts, and management) can view the most relevant information according to their needs.

[0214] Permission Control: Different roles of users can have different access permissions. The module provides a role-based access control mechanism to ensure that different users only view business and technical data related to their responsibilities, ensuring data security.

[0215] ④Data Correlation Analysis:

[0216] This module can automatically correlate business data with technical data. For example, users can analyze the efficiency of order processing by viewing system performance data (such as response time, server load, etc.) for a certain order, and further correlate business information such as order amount, user type, etc. for comprehensive analysis.

[0217] Example scenario: An e-commerce company can use this module to view detailed technical and business data for a certain order processing, including processing time, user experience, payment success rate, etc., to evaluate the bottlenecks in the business process.

[0218] ⑤Historical Data Comparison and Trend Analysis:

[0219] The module supports historical data comparison, allowing users to compare data from multiple time periods to help them identify long-term trends and anomalies in system and business processes. Users can also set thresholds, and the system will automatically generate warnings or recommendations when system performance or business key indicators exceed or fall below the set range.

[0220] Trend analysis function: By analyzing historical data, the module can generate trend charts of business or system performance, helping users identify patterns of system performance changes and predict potential future problems.

[0221] ⑥Report Generation Function:

[0222] The module provides a one-click report generation function, allowing users to generate standardized PDF or Excel reports based on analysis results. These reports can serve as references for daily business or technical team meetings, supporting decision-making and problem review.

[0223] Smart Recommendations and Optimization Suggestions:

[0224] The module also integrates intelligent algorithms to generate optimization suggestions by analyzing derived data. For example, if the system detects frequent operation delays in a certain business process, the module will automatically recommend optimization solutions such as adjusting server configurations or optimizing business logic.

[0225] Example: In an online payment system, if the module detects that the payment success rate of VIP users is low, it can suggest optimizing the performance of the payment gateway or adjusting the priority of the payment process for VIP customers.

[0226] The derived data display module is a powerful tool that integrates high-dimensional data display, interactive analysis, historical data comparison, intelligent recommendation, and other functions. It helps enterprises analyze system operation from both business and technical perspectives, quickly identify problems and provide optimization suggestions, effectively supporting business decisions and improving overall system performance.

[0227] (2) Pragmatic data display module

[0228] The pragmatic data display module displays pragmatic data in natural language form through the prompt engineering. The details are as follows:

[0229] ① Visualization of pragmatic data and natural language generation:

[0230] The module uses multi-dimensional visualization display technology and business data, combined with natural language descriptions generated by large language models, to enable business personnel to easily understand system operation status and business processes.

[0231] Display form: including line charts, column charts, and other technical indicator charts, as well as natural language generated business process descriptions. The natural language generation part explains each step of the business process, allowing technical and business teams to align their understanding.

[0232] Example: In the payment processing scenario, the module can generate a natural language description of "payment confirmation success, order number 123456, total processing time 12 seconds", and display a chart of processing time.

[0233] ② Interactive analysis and real-time update:

[0234] The module supports users to filter, sort and drill down data through interactive interface, and can update analysis results in real time. Business personnel can view the latest business process status of the system and further analyze the details of specific business or technical data.

[0235] Real-time: background data is automatically updated to the interface, ensuring that business personnel can view the latest monitoring data at any time. Users can analyze trends and abnormalities by filtering specific time periods or business processes.

[0236] Interactive functions: allow users to select different dimensions of data for interactive analysis, such as filtering and viewing processing time distribution based on order amount.

[0237] ③ Predictive analysis and optimization suggestions:

[0238] The module generates predictive analysis of business processes based on pragmatic data analysis results and provides business optimization suggestions. By identifying bottlenecks or abnormalities in business processes, the system can automatically recommend adjustment solutions, such as optimizing technical configurations or adjusting business strategies.

[0239] Example: In an order processing system, if the order processing time for a certain type of customer is too long, the module will generate optimization suggestions such as "improve the payment gateway priority of VIP customers to reduce delay".

[0240] ④ Historical data comparison and trend display:

[0241] The module supports users to compare historical data of different time periods, helping to identify long-term trends in business processes. Users can view past analysis results and compare them with current data to evaluate the improvement effect of system performance or business processes.

[0242] Example: Users can view order processing data for the past three months and discover changes in system response speed in a certain period through trend charts, thereby adjusting business strategies or technical configurations.

[0243] The pragmatic data display module combines natural language generation with multi-dimensional visualization to achieve seamless communication between business and technical personnel, breaking down the understanding barriers between technology and business in traditional systems. This allows business personnel to understand complex system operations and business processes through intuitive natural language explanations and visual interfaces without deep technical background. At the same time, the module provides real-time updates, interactive analysis and optimization suggestions, making business decisions more intelligent and efficient, solving the problem of dynamic processing and optimization of business processes in traditional systems.

[0244] In summary, the observability system based on business context enhancement proposed in this embodiment realizes more comprehensive data collection, processing and display without affecting the functions of the original observability system. Enterprises not only maintain the reliability and performance monitoring capabilities of existing systems, but also integrate technical data and business processes deeply with the help of enhanced functions to improve the efficiency of risk control and business response.

[0245] The above description of the embodiments is to facilitate the understanding and use of the invention by ordinary skilled persons in the art. Those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without the need for creative labor. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the scope of protection of the present invention.

Claims

1. A service context enhanced observability system based on an observability architecture, comprising a data collection layer, a data processing layer, a data storage layer and a data display layer, characterized in that, The data collection layer is configured to collect metadata of business requirements and technical requirements, and generate technical monitoring data under a business context based on the metadata; The data processing layer is configured to associate enhanced data with the technical monitoring data under the business context, and generate derived data with unified structure, the enhanced data including external business data and software life cycle data; the data processing layer is further configured to perform semantic analysis on the derived data, and generate pragmatic data, the pragmatic data including business level operation steps and business optimization suggestions in natural language form; The data storage layer is configured to store the derived data and the pragmatic data; and the data display layer is configured to display the derived data and the pragmatic data in a visual form. The data collection layer includes a business context management module, a business context injection module and a dynamic configuration management module, the business context management module is configured to obtain metadata of business requirements and technical requirements, and establish a business data dictionary based on the metadata, the business data dictionary including Span information, Span attributes and log information under a business context; the business context injection module integrates an IDE plug-in, and is configured to inject information in the business data dictionary into a tracking unit and a log of the observability architecture in a structured manner, and obtain technical monitoring data under the business context; and the dynamic configuration management module is configured to dynamically adjust a data collection strategy of the business context management module through a dual mode strategy.

2. The service context augmentation based observability system of claim 1, wherein, The process of establishing the business data dictionary is specifically as follows: extracting a business process name as a name of a tracking unit Span from metadata of business requirements, and establishing a Span information table, the Span information table including the business process name, business meaning and business role; establishing a Span attribute table based on metadata of business requirements and technical requirements, the Span attribute table including field information of a business entity, a life cycle state and an association identifier; extracting association actions between entities from the metadata of business requirements and technical requirements, and establishing a log information table, the log information table including a log number, an association action name and a log template, the log template including associated entities, a behavior type, value changes and additional information.

3. The service context augmentation based observability system of claim 1, wherein, The process of dynamically adjusting the data collection strategy of the business context management module through the dual mode strategy by the dynamic configuration management module is specifically as follows: dynamically adjusting a sampling rate, a filtering rule and a data buffer size of data collection through a configuration interface provided by an observability SDK; judging whether data needs to be collected through an observability function defined based on context information and a predefined rule, the context information including a current business process, a user type and a system load.

4. The service context augmentation based observability system of claim 1, wherein, The data processing layer includes a derived data offline processing module and an intelligent pragmatic analysis processing module, the derived data offline processing module is configured to obtain enhanced data, and associate the enhanced data with preprocessed technical monitoring data under a business context, and generate derived data; The intelligent pragmatic analysis processing module is configured to perform semantic analysis on the derived data by using a large language model, and generate pragmatic data.

5. The service context augmentation based observability system of claim 4, wherein, The specific process of generating the derived data is as follows: Extract the business entity ID in the technical monitoring data under the preprocessed business context, and obtain the external business data accordingly. The preprocessed technical monitoring data under the business context is associated with the external business data by primary key matching, foreign key reference, or a consistent hash-based manner. Extract the key fields in the technical monitoring data under the preprocessed business context, and obtain the software life cycle data based on the release package ID in the key fields. The software life cycle data includes commit hash, configuration file version, and build environment information. The preprocessed technical monitoring data under the business context is associated with the software life cycle data by hash matching, mapping table, or indexing mechanism. According to the hierarchical relationship, correlation relationship, and query requirements of the data, the associated data is uniformly modeled, and a big data processing framework and a distributed search engine are used to generate an index, and finally the derived data is obtained.

6. The service context augmentation based observability system of claim 1, wherein, The data storage layer includes a derived data storage module and a pragmatic data storage module, which are respectively used to receive and store the derived data and the pragmatic data. The derived data storage module first determines the target storage backend of the derived data according to the routing rules, and then further determines the transmission protocol and interface according to the target storage backend, and uses parallel transmission to transmit. The pragmatic data storage module first collects the interaction data of the pragmatic data, and associates it with the corresponding metadata to establish a pragmatic data record. Then, the data storage structure is designed to store the pragmatic data record.

7. The service context awareness enhanced observability system of claim 6, wherein, Both the derived data and the pragmatic data are compressed and selectively encrypted before transmission. During transmission, data integrity and data legality are checked. If the transmission fails, an error log is recorded and automatic retry is performed.

8. The service context augmentation based observability system of claim 6, wherein, If the large language model used to generate a certain segment of pragmatic data is upgraded, the upgraded large language model is used to generate upgraded pragmatic data based on the interaction data of the pragmatic data. The upgraded pragmatic data is compared with the original pragmatic data to generate optimization suggestions for the large language model.

9. The service context augmentation based observability system of claim 1, wherein, The data display layer includes a derived data display module and a pragmatic data display module, which are respectively used to display the derived data and the pragmatic data. The pragmatic data display module displays the pragmatic data in natural language form by prompting engineering.

Images