Observability system based on business context enhancement

Abstract

The invention relates to an observability system based on business context enhancement, which is constructed based on an observability architecture and comprises a data acquisition layer, a data processing layer, a data storage layer and a data display layer, the data acquisition layer is used for acquiring metadata of business requirements and technical requirements and generating technical monitoring data under a business context; the data processing layer is used for associating enhanced data comprising external business data and software life cycle data with technical monitoring data in a business context to generate derivative data; the data processing layer is also used for carrying out semantic analysis on the derivative data and generating pragmatic data, and the pragmatic data comprises business level operation steps in a natural language form and business optimization suggestions; the data storage layer is used for storing derivative data and pragmatic data; and the data display layer is used for displaying derivative data and pragmatic data. Compared with the prior art, deep fusion and comprehensive observability of the technical data and the business process can be realized, and communication obstacles between a technical team and a business team are reduced.

Description

Technical Field

[0001] The present invention belongs to the field of observability technology, and in particular, relates to an observability system based on business context enhancement. Background Art

[0002] Large enterprises need the cooperation of technical teams and business teams to ensure rapid and stable development of the enterprise. The technical team is responsible for maintaining the enterprise's IT system and ensuring its smooth operation, while the business team focuses on market trends, customer needs and business strategies, and uses the enterprise's IT system to handle business issues. However, it is difficult for the technical team to associate a large amount of technical data with actual business impacts and respond to the needs of the business team in a timely manner; the business team also finds it difficult to understand the complex data provided by the technical team and use technical support to handle abnormal business in a timely manner. The main reasons are as follows:

[0003] First, current technical teams widely use observability technology to ensure the stable operation of IT systems. Through logs, performance indicators and tracking information, technical teams can understand the operating status of the system in real time and ensure the efficient operation of the system. However, systems based on traditional observability technology mainly focus on the technical level and lack business context. Data such as business processes, user experience and market response are separated from IT data, making it difficult for departments to collaborate when dealing with risks or problems.

[0004] Second, the business monitoring system of the business team is often independent of the technical monitoring system, and the information island phenomenon is relatively serious. The business department can only rely on result data such as transaction results to judge the health of the business, but lacks process data support. This separation increases the difficulty of timely discovering potential risks, making the troubleshooting and optimization of business problems complicated and inefficient.

[0005] In summary, it is necessary to design an observability system to achieve deep integration and comprehensive observability of technical data and business processes, reduce communication barriers between technical teams and business teams, and simultaneously improve the observability, auditing capabilities, and abnormal response levels of business and technology. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an observability system based on business context enhancement, to achieve deep integration and comprehensive observability of technical data and business processes, to reduce communication barriers between technical teams and business teams, and to enhance the observation, auditing and exception handling capabilities of business-technology integration.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] The present invention provides an observability system based on business context enhancement, which is built based on an observability architecture and includes a data collection layer, a data processing layer, a data storage layer and a data display layer. The data collection layer is used to collect metadata of business requirements and technical requirements, and generate technical monitoring data in the business context based on this; the data processing layer is used to associate enhanced data with technical monitoring data in the business context to generate derived data with a unified structure, and the enhanced data includes external business data and software life cycle data; the data processing layer is also used to perform semantic analysis on the derived data to generate pragmatic data, and the pragmatic data includes business-level operation steps and business optimization suggestions in natural language form; the data storage layer is used to store the derived data and the pragmatic data; the data display layer is used to display the derived data and the pragmatic data in a visual form.

[0009] Furthermore, the data acquisition layer includes a business context management module, a business context injection module and a dynamic configuration management module. The business context management module is used to obtain metadata of business requirements and technical requirements, and establish a business data dictionary based on this. The business data dictionary includes Span information, Span attributes and log information under the business context; the business context injection module integrates an IDE plug-in to inject the information in the business data dictionary into the tracking unit and log of the observability architecture in a structured manner to obtain technical monitoring data under the business context; the dynamic configuration management module is used to dynamically adjust the data acquisition strategy of the business context management module through a dual-mode strategy.

[0010] Furthermore, the process of establishing the business data dictionary is as follows:

[0011] Extract the business process name from the business requirements as the name of the tracking unit Span, and establish a Span information table, wherein the Span information table includes the business process name, business meaning and business function;

[0012] Based on business requirements and technical requirements, a Span attribute table is established, wherein the Span attribute table includes field information, life cycle status, and association identifier of a business entity;

[0013] The associated actions between entities are extracted from the business requirements and technical requirements, and a log information table is established. The log information table includes a log number, an associated action name and a log template. The log template includes an associated entity, a behavior type, a value change and additional information.

[0014] Furthermore, the data collection strategy of the business context management module is loaded and applied through the CI / CD component, and the process in which the dynamic configuration management module dynamically adjusts the data collection strategy of the business context management module through the dual-mode strategy is specifically as follows:

[0015] Dynamically adjust the sampling rate, filtering rules, and data buffer size of data collection through the configuration interface provided by the observability SDK;

[0016] Whether data needs to be collected is determined by an observability function defined based on context information and predefined rules. The context information includes the current business process, user type, and system load.

[0017] Furthermore, the data processing layer includes a derived data offline processing module and an intelligent pragmatic analysis processing module. The derived data offline processing module is used to obtain enhanced data and associate it with the pre-processed technical monitoring data in the business context to generate derived data; the intelligent pragmatic analysis processing module is used to perform semantic analysis on the derived data through a large language model to generate pragmatic data.

[0018] Furthermore, the specific process of generating the derivative data is as follows:

[0019] Extracting the business entity ID from the preprocessed technical monitoring data in the business context, obtaining the external business data based on the business entity ID, and associating the preprocessed technical monitoring data in the business context with the external business data through primary key matching, foreign key reference, or consistent hashing;

[0020] Extract key fields from the preprocessed technical monitoring data in the business context, and obtain software lifecycle data based on the release package ID in the key fields, wherein the software lifecycle data includes submission hash, configuration file version, and build environment information; associate the preprocessed technical monitoring data in the business context with the software lifecycle data through hash matching, mapping table, or index mechanism;

[0021] According to the hierarchical relationship, association relationship and query requirements of the data, the associated data is uniformly modeled, and the big data processing framework and distributed search engine are used to generate indexes to finally obtain derived data.

[0022] Further, the data storage layer includes a derivative data storage module and a pragmatic data storage module, which are respectively used to receive and store the derivative data and the pragmatic data;

[0023] The derived data storage module first determines the target storage backend of the derived data according to the routing rules, and then further determines the transmission protocol and interface according to the target storage backend, and transmits the data in a parallel transmission mode;

[0024] The pragmatic data storage module first collects the interaction data of pragmatic data, associates it with corresponding metadata, and establishes pragmatic data records; then designs a data storage structure to store the pragmatic data records.

[0025] Furthermore, the derivative data and the pragmatic data are compressed and selectively encrypted before transmission; during the transmission process, data integrity and data legality are checked; if the transmission fails, an error log is recorded and a retry is automatically performed.

[0026] Furthermore, if the large language model that generates a certain segment of pragmatic data is upgraded, the upgraded large language model is used to generate upgraded pragmatic data based on the interactive data of the pragmatic data, and the upgraded pragmatic data is differentially analyzed with the original pragmatic data to generate optimization suggestions for the large language model.

[0027] Furthermore, the data display layer includes a derived data display module and a pragmatic data display module, which are respectively used to display the derived data and the pragmatic data. The pragmatic data display module displays the pragmatic data in a natural language form through a prompting project.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The present invention proposes an observability system based on business context enhancement, which is built based on the observability architecture and includes a data collection layer, a data processing layer, a data storage layer and a data display layer. The data collection layer is used to collect metadata of business requirements and technical requirements, and generate technical monitoring data in the business context based on this. The technical team and the business team can share this set of data, which greatly reduces communication barriers and improves the speed of problem troubleshooting and business optimization; the data processing layer is used to deeply associate enhanced data such as external business data and software life cycle data with technical monitoring data in the business context, and generate derivative data with a unified structure, comprehensiveness and cross-dimensionality, which can be used for subsequent large-scale It provides strong support for data analysis, business insights and predictions, greatly improving the system's analysis, auditing and early warning capabilities in cross-domain scenarios; the data processing layer is also used to perform semantic analysis on derived data and generate pragmatic data. Pragmatic data includes business-level operation steps and business optimization suggestions in natural language form, which can improve the sophistication and accuracy of analysis and enable business teams to understand complex business processes and system performance more intuitively, thereby improving the efficiency of business decision-making and optimization; the data display layer is used to display derived data and pragmatic data in a visual form, which can intuitively observe complex technical monitoring data, achieve seamless communication between business and technical personnel, and make business decisions more intelligent and efficient.

[0030] 2. The present invention establishes a business data dictionary based on the metadata of business needs and technical needs. The business data dictionary includes Span information, Span attributes and log information in the business context. It can deeply integrate information such as business process name, attribute information and associated actions between entities into the technical monitoring system, realize the deep integration of business context and technical monitoring and the standardized management of metadata information, and support fine-grained analysis in complex business scenarios.

[0031] 3. In the present invention, the data collection strategy of the business context management module is loaded and applied through the CI / CD component, and the data collection strategy of the business context management module is dynamically adjusted through the dual-mode strategy. Specifically, the sampling rate, filtering rules and data buffer size of data collection are dynamically adjusted through the configuration interface provided by the observability SDK; the need to collect data is determined by the observability function defined based on context information and predefined rules, and the context information includes the current business process, user type and system load; the above design can minimize unnecessary data and optimize system performance while ensuring the collection of key data. The policy update automatically takes effect with the CI / CD process, which can reduce configuration errors and improve system stability.

[0032] 4. The present invention specifically generates derived data through the following steps: extracting the business entity ID in the preprocessed technical monitoring data in the business context, obtaining external business data based on it, and associating the preprocessed technical monitoring data in the business context with the external business data through primary key matching, foreign key reference or consistent hashing; extracting the key fields in the preprocessed technical monitoring data in the business context, and obtaining software life cycle data based on the release package ID in the key fields, the software life cycle data including submission hash, configuration file version and build environment information; associating the preprocessed technical monitoring data in the business context with the software life cycle data through hash matching, mapping table or index mechanism; finally, according to the hierarchical relationship, association relationship and query requirements of the data, the associated data is uniformly modeled, and an index is generated using a big data processing framework and a distributed search engine; the above design can achieve deep association and enrichment of observability data, business data, and software life cycle data, improve the system's anomaly detection efficiency and prediction capabilities, and is suitable for complex new business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The diagram below is a structural diagram of the observability system enhanced based on business context. DETAILED DESCRIPTION

[0034] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0035] Example:

[0036] In traditional observability tools, technical data and business data are usually separated, which makes it difficult for technical personnel to quickly understand the direct impact of system failures on the business. This embodiment introduces business context into the observability system and provides an observability system enhanced based on business context, which closely integrates technical data with business processes. The system is built based on the observability architecture (such as OpenTelemetry) and does not affect the original performance monitoring, error tracking, indicator display and other functions of the observability architecture. You can continue to use logs, indicators and tracking functions, as well as existing tools such as Prometheus, Jaeger, Grafana, etc. Figure 1 As shown in Figure 1, the system includes a data collection layer, a data processing layer, a data storage layer, and a data display layer. The improvement lies in injecting business processes, entity information, and operation context into technical monitoring data, so that observable data can not only display the system operation status, but also be closely integrated with business processes to support real-time monitoring and optimization of the business. The following will introduce the structure and function of each layer in detail.

[0037] Ⅰ. Data Collection Layer

[0038] The data collection layer includes a business context management module, a business context injection module, and a dynamic configuration management module. The business context management module is used to obtain metadata of business requirements and technical requirements, and establish a business data dictionary based on this. The business data dictionary includes Span information, Span attributes, and log information in the business context. The business context injection module integrates an IDE plug-in to inject the information in the business data dictionary into the tracking unit and log of the observability architecture in a structured manner to obtain technical monitoring data in the business context. The dynamic configuration management module is used to dynamically adjust the data collection strategy of the business context management module through a dual-mode strategy. The specific description of each module is as follows:

[0039] (1) Business context management module

[0040] The context management module establishes a unified business data dictionary by standardizing the metadata of business requirements and technical requirements. The specific process of establishing the business data dictionary is as follows:

[0041] S111. Extract and manage the business process name as the tracking unit name.

[0042] Using the parsing algorithm, extract the business process name (such as "contract creation", "product classification update", etc.) from the metadata of the business requirements as the name of the tracking unit Span and record it in the Span information table. The core data of the Span information table includes the business process name (the name of the Span, corresponding to the name of the business process or process step), the list of related attributes (the list of attribute names related to the Span and the business role) and the description (a detailed description of the Span, explaining its business meaning and role). This step can make each Span have clear business semantics, which is convenient for the technical team and the business team to quickly align the system status.

[0043] S112. Define and manage the declaration of Span attribute metadata.

[0044] Based on the metadata of business requirements and technical requirements, a span attribute table is established. The core data of the span attribute table includes the attribute name (the name of the span attribute, which identifies a specific business or technical indicator) and the attribute description (a detailed description of the attribute, explaining its meaning and purpose). Through the span attribute table, the definition of each attribute can be standardized to ensure data consistency. These attributes may include field information, life cycle status, and associated identifiers of business entities.

[0045] S113. Manage and record the associated action categories and templates between entities as log content.

[0046] Extract the associated actions between entities from the metadata of business and technical requirements, and establish a log information table. The core data of the log information table includes the log number (the unique identifier of the log record), the associated action name (such as "product category update", "order status change", etc.) and the log template (defining the format and content of the log, including the associated entity, behavior type, value change, additional information, etc.). The log information table can provide more fine-grained business observability, helping technical and business teams to quickly locate problems, audit business processes, strengthen monitoring and analysis of business operations, and improve risk control capabilities.

[0047] The business context management module deeply integrates the business process name, attribute information (including metadata of business entities, lifecycles and other fields) and inter-entity related actions into the monitoring system, realizing the deep integration of business context and technical monitoring, which can enhance business observability, improve problem location efficiency, and promote business and technical collaboration. By standardizing the management of metadata information, establishing a complete business data dictionary, and supporting fine-grained analysis in complex business scenarios, the system maintainability can be improved, the system scalability can be enhanced, and the data utilization rate can be increased.

[0048] (2) Business context injection module

[0049] The context injection module is responsible for injecting business process information, entity information, and associated actions into the tracking unit (Span) and log (log) of the observability system in a structured manner. An important function of this module is to provide IDE plug-ins to assist developers in referring to the standards of the context management module during the coding process and conveniently complete the injection of context information. In this way, the close integration of business requirements and technical observability systems is ensured to form a digital twin of the system during operation. The data processing process within this module is as follows:

[0050] S121. Name the Span (tracing unit) based on the business requirement description.

[0051] When naming each span, use the business process or sub-process name from the span information table of the context management module. The specific operation is: the developer manually calls the API of the observability SDK in the code to create and name the span. When coding, refer to the standards provided by the context management module to name the span name and ensure that it complies with the standard naming of the business process. The business context injection module provides an IDE plug-in that automatically reads the span information table of the context management module. In the IDE, you can easily select the appropriate span name through the drop-down list to improve the development experience and reduce manual input errors.

[0052] Through the above steps, technical and business personnel can intuitively understand the specific business processes being processed by the system during debugging and analysis, and promote collaboration between business and technical teams.

[0053] S122. Record entity classification, entity information, and cross-domain association information in the Span attribute.

[0054] When coding, developers inject key information of business entities and related data of cross-domain associations into the attributes of Span in the form of key-value pairs. The specific operation is: Use the API of the Observability SDK to manually add Span attributes in the code. Refer to the Span attribute table of the context management module to ensure that the attribute name meets the standard. The IDE plug-in of the business context injection module automatically reads the Span attribute table and provides a drop-down list when coding, which makes it convenient for developers to select attribute names and corresponding variable values. The plug-in can also prompt the attributes that need to be filled in to ensure the completeness and accuracy of the attributes. For cross-domain association information, such as release package version, hash number, etc., inject global association information into the root Span of Trace or related Span.

[0055] By recording key information about business entities and cross-domain associations, observability data is enriched. The system can not only capture key contextual information in business processes, but also perform association analysis across multiple business domains at the data processing layer, providing strong support for in-depth business analysis, troubleshooting, and cross-system fault tracking.

[0056] S123. Record the association actions between entities into a log.

[0057] Developers record the associated actions between different entities in the business process in the code and generate detailed log information. The specific operation is as follows: Use the API of the observability SDK to manually create log records. Refer to the log information table of the context management module to ensure that the format and content of the log meet the standards. The IDE plug-in of the context injection module automatically reads the log information table and provides a drop-down list. Developers can select the appropriate log template and fill in the necessary variable values. The plug-in can automatically generate log code snippets to further simplify the development process.

[0058] IDE plug-ins are used to assist developers in conveniently injecting business context information during the coding process. The plug-in works in the developer's coding environment and provides developers with automated support such as auto-completion, input hints, and code generation by interacting with the metadata of the context management module. The main purpose of the plug-in is to automatically complete method names and parameters according to the context during the development process, provide relevant property and method suggestions during the coding process, and automatically generate code snippets such as chain calls, so as to seamlessly introduce standard metadata in the context management module into the code and ensure the accuracy and consistency of business context information.

[0059] The input of the IDE plug-in includes the metadata of the context management module (including the Span information table, Span attribute table and log information table) and the developer's coding environment information (including the currently edited source code file and location, project configuration and dependencies). The plug-in mainly implements the following functions: ① Integration with the context management module: Obtain the latest metadata information from the context management module at startup or regularly to ensure that developers are using the latest standards. It supports manual refresh and automatic synchronization. The obtained metadata can be cached locally to reduce dependence on remote services and improve response speed; ② User interface interaction: automatic completion and drop-down list, that is, when the developer calls the relevant method of the observability SDK in the code, the plug-in automatically provides a drop-down list of available Span names, attributes and log templates; input prompts and verification, that is, when the developer enters parameters, the plug-in provides input prompts, including the meaning of the parameters, the expected data type and examples, identifies the required and optional items, and prompts for missing required items when necessary; code snippet generation, that is, the plug-in can automatically generate the corresponding code snippet based on the selected Span name, attribute and log template, insert it into the current cursor position, and generate The code follows the coding standards and style of the project, reducing the workload of manual writing; it can automatically generate observable code snippets such as chain calls; ③ Integration with Observability SDK: API mapping, that is, the plug-in is pre-configured with the API mapping of the Observability SDK, and knows how to call methods to create Span, add attributes, and record logs. It supports multiple programming languages ​​​​and frameworks to adapt to different development environments; context management, that is, the plug-in assists in processing the context transfer of Span, ensures that the created Span is correctly associated with the parent Span, and provides auxiliary tools to help developers manage context objects to avoid context loss or association errors; ④ Configuration and customization: Developers can configure synchronization frequency, display options, shortcut keys, etc. through the plug-in settings interface, support enabling or disabling specific functions, such as automatic code generation, input verification, etc. The plug-in allows configuration of the project's coding standards, log format, and naming rules to ensure that the generated code meets project requirements.

[0060] The output of the IDE plug-in includes generated code snippets, prompt information during the development process, and metadata update logs. Code snippets include Span creation code, Span property setting code, and logging code. Prompt information includes errors and warnings, as well as optimization suggestions. The metadata update log includes the synchronization log and the log of the plug-in recording and synchronizing metadata with the context management module, which is convenient for developers to view metadata updates.

[0061] Through the above design, the IDE plug-in can act as a bridge between the business context management module and developers, effectively connecting the metadata of the business context management module and the coding practices of developers. The plug-in's automatic completion and code generation functions significantly improve development efficiency, allowing developers to focus on the implementation of business logic; input hints and verification functions help developers find and correct errors in a timely manner to ensure code quality; unified naming and format specifications facilitate subsequent data processing, analysis, and display, ensuring that the injection of business context information into the code complies with the standards of the context management module, maintaining data consistency and accuracy, and promoting collaboration within the team.

[0062] Example of use:

[0063] ① Create Span: The developer creates a new Span in the code. After entering startSpan, the plug-in automatically pops up a drop-down list to display the available business process names. For example, after selecting order creation, the plug-in generates the code: Spanspan = tracer.startSpan ("Order creation");

[0064] ②Add Span attributes: After creating Span, enter setAttribute, and the plug-in prompts the available attribute names, such as order ID and user ID. For example, select order ID, the plug-in automatically fills in the code and prompts to fill in the corresponding variable value: span.setAttribute("Order ID",orderId);

[0065] ③Record logs: In the business logic, you need to record a related action log. Enter logEvent, and the plug-in provides a choice of log templates. For example, select the order status update template, and the plug-in generates the code: Span.addEvent ("order status update", attributes);

[0066] The plug-in prompts for required properties, such as old status and new status. After the developer fills in the properties, logging is completed.

[0067] (3) Dynamic Configuration Management Module

[0068] The amount of observable data is huge, and the cost of data collection and storage is high. In order to optimize system performance and reduce costs, it is necessary to dynamically adjust the data collection strategy. This module implements dynamic adjustment of data collection strategy based on the following two methods:

[0069] 1) Take effect through the Observability SDK.

[0070] Through the configuration interface provided by the observability SDK, the sampling rate, filtering rules, and data buffer size of data collection can be dynamically adjusted. This method directly affects the original configuration of the observability system and implements global policy adjustments.

[0071] 2) Through custom observability functions

[0072] Before data collection, whether data needs to be collected is determined by using observability functions defined based on context information (including current business processes, user types, and system load) and predefined rules.

[0073] The above methods can meet diverse business needs and adapt to different scenarios. In key business scenarios, they can ensure the collection of sufficient fine-grained data to support problem troubleshooting and performance analysis. At the same time, they can minimize unnecessary data collection and storage and optimize system performance.

[0074] Integrating the update of data collection strategies with the continuous integration and continuous delivery (CI / CD) process ensures that strategy changes take effect when the application is redeployed. This allows for rapid response to business changes, shortens the strategy adjustment cycle, reduces configuration errors, and improves system stability.

[0075] II. Data Processing Layer

[0076] The data processing layer includes a derivative data offline processing module and an intelligent pragmatic analysis processing module. The derivative data offline processing module is used to obtain enhanced data (including external business data and software life cycle data) and associate it with the pre-processed technical monitoring data in the business context to generate derivative data. The intelligent pragmatic analysis processing module is used to perform semantic analysis on the derivative data through a large language model to generate pragmatic data. The pragmatic data includes business-level operation steps and business optimization suggestions in natural language form. The specific description of each module is as follows:

[0077] (1) Derivative data offline processing module

[0078] This module enriches observable data, deeply associates technical data with business data, software lifecycle data, etc., generates cross-dimensional derivative data, and supports intelligent analysis and prediction in complex business scenarios. The module processes data as follows:

[0079] S211. Data collection and preprocessing

[0080] Acquire external business system data (including business data from CRM, ERP, supply chain management systems, etc.), software lifecycle data (including commit information in the code base, release package IDs of CI / CD systems, version information of configuration management systems, etc.) and technical monitoring data in the business context. Clean, format and standardize the collected data to ensure data quality and consistency.

[0081] S212, Data Enrichment

[0082] Association of observability data with software lifecycle data: First, extract key fields (including release package ID, Trace ID, Span ID) from the preprocessed technical monitoring data in the business context; then, call the API of the CI / CD system based on the release package ID to obtain software lifecycle data (including commit hash, configuration file version, and build environment information); second, associate the preprocessed technical monitoring data in the business context with the software lifecycle data through hash matching, mapping table or index mechanism. The following is an example: When the root Span of a Trace contains a release package ID, the corresponding commit hash and related build information can be obtained from the CI / CD system through this ID.

[0083] Association of observability data with external business system data: First, extract the business entity ID (such as user ID, order ID, product ID, etc.) from the pre-processed technical monitoring data in the business context; then, obtain external business data through API or data warehouse; secondly, associate the pre-processed technical monitoring data in the business context with the external business data through primary key matching, foreign key reference or consistent hashing. The following is an example: Get detailed information about the user from the CRM system through the user ID, such as user level, industry, historical purchase records, etc.

[0084] S213. Data index generation and storage

[0085] First, consider the hierarchical relationship, association relationship and query requirements of the data, and design a unified data model to adapt to the multi-dimensional data structure, including technical indicators, business attributes and life cycle information. Secondly, use big data processing frameworks (such as Hadoop, Spark) and distributed search engines (such as Elasticsearch) to generate indexes, index the enriched data, and obtain structured derivative data (including span hierarchical relationships, time-consuming information, business attributes, etc.), supporting real-time query and complex data aggregation analysis. Indexing methods include inverted index (establishing inverted indexes for key fields such as user ID, order ID, TraceID, error code, etc. to support fast query), shard index (sharding data by time, business module or other dimensions, distributed storage and processing, and improving concurrent query capabilities) and data compression (using data compression technology, such as Parquet format, column storage, to optimize storage space and query performance), all of which can build efficient big data indexes. Subsequently, the index and data are stored in a high-performance storage system to support real-time query and complex analysis.

[0086] Through the derivative data offline processing module, the original observability data not only records the technical status of the system, but is also enriched with business and lifecycle data to form more comprehensive, cross-dimensional derivative data. This module describes the input, output, and processing of data enrichment in detail to achieve achievable standards. It provides strong support for subsequent big data analysis, business insights, and predictions, greatly enhancing the system's intelligent analysis and risk warning capabilities.

[0087] Here are three usage examples:

[0088] Example 1: Error location

[0089] When the system detects an error, it finds the related span through the trace ID, extracts the release package ID in the span, associates it with the CI / CD system, and obtains the corresponding commit hash. Through the commit hash, the specific code changes, developers, and code review information are located. Through the above processing, technicians can quickly locate the source of the problem and speed up problem troubleshooting.

[0090] Example 2: User behavior analysis

[0091] If the business team wants to analyze the usage experience of VIP users, first, extract the user ID from the observable data, associate it with the CRM system, and obtain the user level. Then, in the enriched data set, query the VIP user's operation track, response time, error situation, etc. Through the above processing, the business team can optimize services for VIP users and improve satisfaction.

[0092] Example 3: Risk warning

[0093] If the system needs to detect possible fraudulent behavior, first, it associates the user's historical behavior data through the user ID and analyzes the user's behavior patterns in multiple business processes; then, it uses the enriched multi-dimensional data and applies machine learning algorithms to identify abnormal behavior.

[0094] (2) Intelligent pragmatic analysis and processing module

[0095] The intelligent pragmatic analysis and processing module performs semantic analysis and processing on the collected raw observability data, business context information, and structured span and log information to generate natural language interpretation, analysis reports, and business optimization suggestions for business personnel to help business decision-making and problem location. The data processing process within the module is as follows:

[0096] S221. Generate semantically rich LLM input

[0097] Through the data interface or message queue, the structured derivative data (including the hierarchical relationship of Span, time-consuming information, business attributes, etc.) after preprocessing (including data parsing, hierarchical organization, data cleaning and data structuring, etc.) is pushed to the intelligent pragmatic data analysis module and converted into an input format understandable to LLM, such as text description or specific input template, which organizes the data in a human-readable form and contains necessary contextual information.

[0098] Example: Construct a description to reflect the hierarchical information of Span:

[0099] The customer with user ID 12345 completed a payment process, which took a total of 10.45 seconds. The specific operations included:

[0100] ①Order creation (3.12 seconds)

[0101] -Verify inventory (1.00 seconds): Check if the product is in sufficient stock.

[0102] -Save order (2.12 seconds): Save the order information to the database.

[0103] ② Payment confirmation (5.89 seconds)

[0104] -Third-party payment interface call (5.00 seconds): interact with the payment platform to process payment.

[0105] -Update order status (0.89 seconds): Update the order status to paid.

[0106] ③Customer notification (1.44 seconds)

[0107] -Send SMS notification (0.50 seconds): Send a successful payment SMS to the customer's mobile phone.

[0108] -Update notification record (0.94 seconds): record the notification sending status.

[0109] This description clearly reflects the hierarchical structure of Span, showing the time and description of each operation step and its sub-steps, which facilitates in-depth analysis of each link of complex business processes and discovers potential problems and optimization points.

[0110] S222: Calling the Large Language Model (LLM) for semantic analysis and content generation

[0111] First, use a pre-trained large language model (such as GPT-4), input semantically rich text, set appropriate prompts, and guide LLM to analyze and generate; second, design prompt words to allow LLM to understand the input content and analyze user behavior, system performance, potential problems, etc.; third, based on the analysis results, generate natural language explanations, analysis reports or business optimization suggestions. It can not only improve the sophistication and accuracy of the analysis, but also enable business personnel to understand complex business processes and system performance more intuitively, improving the efficiency of business decision-making and optimization. For the example mentioned in S221, the corresponding LLM generates the following output:

[0112] The payment process for user 12345 took a total of 10.45 seconds. The following is a detailed analysis:

[0113] ①Order creation (3.12 seconds)

[0114] -Verify Inventory (1.00 sec): Normal time taken.

[0115] -Save order (2.12 seconds): It takes a little longer and the database write performance may need to be optimized.

[0116] ② Payment confirmation (5.89 seconds)

[0117] -Third-party payment interface call (5.00 seconds): It takes a long time. It may be that the third-party payment platform responds slowly. It is recommended to communicate with the payment platform for optimization.

[0118] -Update order status (0.89 seconds): normal time consumption.

[0119] ③Customer notification (1.44 seconds)

[0120] -Sending SMS notification (0.50 seconds): normal time consumption.

[0121] -Update notification record (0.94 seconds): normal time consumption.

[0122] suggestion:

[0123] -Optimize order saving process and check database performance.

[0124] -Contact the third-party payment platform to understand the response of the payment interface and seek optimization solutions.

[0125] -Overall, the time spent in the payment process is mainly concentrated in the payment confirmation stage, which requires special attention.

[0126] Through the above processing, the intelligent pragmatic analysis processing module makes full use of the hierarchical information of Span and generates semantically rich input for the large language model to perform in-depth semantic analysis and content generation. This method not only improves the sophistication and accuracy of the analysis, but also enables business personnel to more intuitively understand complex business processes and system performance, improving the efficiency of business decision-making and optimization.

[0127] III. Data Storage Layer

[0128] The data storage layer includes a derived data storage module and a pragmatic data storage module, which are used to receive and store derived data and pragmatic data respectively. The specific description of each module is as follows:

[0129] (1) Derivative Data Storage Module

[0130] In order to support diverse business needs and application scenarios, it is necessary to efficiently store derived data in different data backends for use by various systems and applications. The data processing process in the derived data storage module is as follows:

[0131] S311. Data formatting and standardization

[0132] Unified data format: Ensure that the derived data conforms to the format and structure required by each target storage backend, so that the derived data can be seamlessly shared and retrieved in different storage systems. Although the standardized JSON format is used, different storage systems may have specific field requirements or restrictions, requiring appropriate format conversion or field mapping.

[0133] Field mapping and data cleaning: Map and clean the fields in the data to ensure the accuracy and completeness of the data. For example, unify the date format and filter out null values ​​or invalid data.

[0134] S312, Data Distribution

[0135] Determine the target storage backend of the derived data based on the routing rules to meet the multi-dimensional analysis requirements in different business scenarios. For example, for real-time monitoring data, export to Splunk to meet real-time query and monitoring requirements; for full-text retrieval requirements, export to Elasticsearch / OpenSearch, which supports full-text search and complex queries; for OLAP analysis requirements, export to Doris, which supports multi-dimensional analysis and reporting.

[0136] S313, Data transmission

[0137] The transmission protocol and interface (such as HTTP, TCP, dedicated client, etc.) are further determined according to the target storage backend, and an efficient parallel transmission mechanism is used for transmission. Data is compressed before transmission to reduce bandwidth usage, and sensitive data is encrypted for transmission to ensure data security. During the transmission process, the data is verified (such as consistency verification and transaction control) to ensure that the data has not been tampered with or damaged, thereby ensuring the data consistency of each storage backend. When data transmission fails, retries are automatically performed to ensure the reliability of data transmission. At the same time, transmission errors are recorded in detail for subsequent analysis and processing.

[0138] S314. Data storage

[0139] If the storage backend is Splunk, use the API or data receiver provided by Splunk to send data to Splunk. You may need to adjust the data to a specific format to meet Splunk's indexing and search requirements. If the storage backend is Elasticsearch / OpenSearch, use Elasticsearch's Bulk API to implement batch import and indexing of data, define appropriate index structure and mapping, and optimize query performance. If the storage backend is Doris: Use Doris's Stream Load or Broker Load function to import data into Doris, configure Doris's table structure, define column types and partitioning strategies, meet analysis requirements, and provide reliable data support for business decisions.

[0140] (2) Pragmatic Data Storage Module

[0141] This module is used to systematically store all raw data, requests (Prompt), return results, and related metadata generated during intelligent pragmatic analysis. By comprehensively recording the interaction information and analysis process of the large language model (LLM), it can better support large model upgrades, retrieval enhancement (RAG) and other model improvements and tests. This not only helps to continuously optimize model performance, but also provides a solid data foundation for the retrospection, reanalysis, and advanced comprehensive analysis of historical analysis, promoting the accuracy of business decisions.

[0142] The input data of the pragmatic data storage module includes LLM interaction raw data and related metadata. LLM interaction raw data includes: ① Request (Prompt): text request sent by the user or system to the large language model, including business context, task instructions and related data; ② Return result (Response): the reply of the large language model to the request, including the generated analysis report, suggestions or answers. Related metadata includes: ① Model version information: the version number or identifier of the large language model used; ② Request timestamp: the date and time when the interaction occurred; ③ Requester identity: the user ID, role or system module name that initiated the request; ④ Session ID: if there are multiple rounds of dialogue, the identifier associated with the same session; ⑤ Business scenario: the associated business process name, task type or operation category; ⑥ Associated entity information: the identification and related information of the business entity (such as user ID, order ID). The following is an example input:

[0143] Request (Prompt):

[0144] Task: Please analyze the data of the following user payment process, identify possible performance bottlenecks, and provide optimization suggestions.

[0145] data:

[0146] The customer with user ID 12345 completed a payment process, which took a total of 10.45 seconds. The specific operations included:

[0147] ①Order creation (3.12 seconds)

[0148] -Verify inventory (1.00 seconds): Check if the product is in sufficient stock.

[0149] -Save order (2.12 seconds): Save the order information to the database.

[0150] ② Payment confirmation (5.89 seconds)

[0151] -Third-party payment interface call (5.00 seconds): interact with the payment platform to process payment.

[0152] -Update order status (0.89 seconds): Update the order status to paid.

[0153] ③Customer notification (1.44 seconds)

[0154] -Send SMS notification (0.50 seconds): Send a successful payment SMS to the customer's mobile phone.

[0155] -Update notification record (0.94 seconds): record the notification sending status.

[0156] Based on the above data, please generate a detailed analysis report, point out the steps that take a long time, and provide corresponding optimization suggestions.

[0157] ·Return result (Response):

[0158] Analytical Report:

[0159] The payment process for user 12345 took a total of 10.45 seconds. The following is a detailed analysis:

[0160] ①Order creation (3.12 seconds)

[0161] -Verify Inventory (1.00 sec): Normal time taken.

[0162] -Save order (2.12 seconds): It takes a little longer. It is recommended to optimize the database write performance.

[0163] ② Payment confirmation (5.89 seconds)

[0164] -Third-party payment interface call (5.00 seconds): It takes a long time, which may be due to the slow response of the payment platform. It is recommended to negotiate with the payment platform for optimization.

[0165] -Update order status (0.89 seconds): normal time consumption.

[0166] ③Customer notification (1.44 seconds)

[0167] -Sending SMS notification (0.50 seconds): normal time consumption.

[0168] -Update notification record (0.94 seconds): normal time consumption.

[0169] Overall recommendations:

[0170] -Optimize order saving process and check database performance.

[0171] -Communicate with third-party payment platforms to optimize payment interface response time.

[0172] Related metadata:

[0173] Model version: GPT-4

[0174] Request timestamp: 2023-10-15 14:32:10

[0175] Requester Identity: User ID 67890 (Business Analyst)

[0176] Session ID: session_abc123

[0177] Business scenario: Payment process performance analysis

[0178] Related entity information: User ID 12345, Order ID O123456

[0179] The data processing process in the pragmatic data storage module is as follows:

[0180] S321. Data collection and archiving

[0181] Capture and save every interaction with the large language model, including requests (Prompt) and returned results (Response), and associate relevant metadata with the corresponding interaction data to form a complete record.

[0182] S322, Data storage structure design

[0183] Use a database suitable for high-concurrency writing and querying, such as a NoSQL database (MongoDB, Cassandra) or a relational database (MySQL, PostgreSQL). Data model design includes main table and index design.

[0184] Main table: LLM_Interaction

[0185] Fields:

[0186] interaction_id (primary key)

[0187] prompt_text(request content)

[0188] response_text (return result)

[0189] model_version (model version)

[0190] timestamp (request timestamp)

[0191] requester_id (requester identity)

[0192] session_id

[0193] business_context

[0194] entity_info (related entity information, JSON format)

[0195] Index design: Create indexes on commonly used query fields, such as timestamp, requester_id, model_version, and business_context.

[0196] Perform integrity checks on data to ensure that requests and return results are successfully saved to avoid data loss. Desensitize or encrypt sensitive information for storage to meet data security and privacy protection requirements. Ensure that data storage complies with relevant laws and regulations, such as GDPR, Cybersecurity Law, etc.

[0197] After the large language model used by the system is upgraded (for example, from GPT-4 to GPT-O1), it is necessary to re-analyze and evaluate the performance improvement of the new model in business analysis. The processing steps are as follows: First, extract all interaction records in the past month from the pragmatic data storage module, including prompts and related metadata; then, use GPT-O1 to regenerate the return results for these prompts; then, store the new return results together with the corresponding prompts and metadata, and mark the model version as GPT-O1; finally, use the difference analysis tool to summarize the performance improvement of the new model in different business scenarios, and point out specific improvements and areas that need optimization.

[0198] In intelligent pragmatic analysis, RAG technology is used to improve the accuracy of the model's answers in combination with the company's internal knowledge base. For model improvements based on retrieval enhancement (RAG), the external knowledge base content used, the reference materials retrieved, and RAG-related metadata are recorded in each interaction; when the knowledge base is updated or the RAG strategy is adjusted, the stored data is used to re-interact, the effect of the new strategy is evaluated, the quality of the model's answers under the new and old RAG strategies is compared, and the knowledge base content or RAG strategy is adjusted to further improve the model performance and answer accuracy.

[0199] This module implements comprehensive storage of all original data, requests, return results, and business contexts during the LLM interaction process. It uses the stored historical data to re-analyze and compare results after the large language model is upgraded or the RAG is enhanced. By completely preserving the original data and metadata, each analysis process can be traced, ensuring the integrity and auditability of the data and supporting the continuous optimization of the model.

[0200] IV. Data presentation layer

[0201] The data display layer includes a derived data display module and a pragmatic data display module, which are used to display derived data and pragmatic data respectively. The specific description of each module is as follows:

[0202] (1) Derivative data display module

[0203] The derived data display module is the UI front end of the derived data in the system, responsible for presenting the enriched and processed derived data to users in an intuitive and user-friendly way. This module provides a high-dimensional data visualization interface, integrates business data and technical data, helps users deeply analyze system performance, business processes and potential optimization opportunities, and supports business decision-making and problem location.

[0204] Data from the derived data storage module, including all structured business and technical data, is transmitted to the derived data display module through a standardized interface to ensure that the latest business and technical monitoring results are always displayed. The specific instructions are as follows:

[0205] ①Multi-dimensional data display:

[0206] This module integrates technical and business data and presents them to users in a multi-dimensional form. For example, technical data (such as response time, system load) and business data (such as order status, user information) will be presented in the same interface in charts, tables or other visual forms. Users can select different data dimensions for filtering and combination, helping them quickly locate business or technical problems.

[0207] Visualization tools: The module supports a variety of visualization tools, such as line charts, pie charts, bar charts, heat maps, etc., to help users intuitively understand data trends and relationships.

[0208] Example: For performance monitoring of a business operation, users can view the relationship between the processing time of each order and key business indicators, such as order amount, user type, etc.

[0209] ②Interactive data analysis:

[0210] Users can filter, sort and drill-down data in real time through the module. The interactive function allows users to drill down from a macro perspective to specific business operations. For example, users can gradually refine the performance data or user operation behavior of a specific business entity from the monitoring view of the overall business process.

[0211] Real-time data update: This module integrates real-time data update function, allowing users to view the latest monitoring data without refreshing the page. This ensures that users can make quick responses and business adjustments based on the latest system status.

[0212] ③Customized display:

[0213] The module allows users to customize data display templates according to business needs. Users can choose specific data fields, visualization styles, and display priorities to ensure that users at different levels (such as technicians, business analysts, and management) can view the most relevant information according to their needs.

[0214] Permission control: Users with different roles can have different access rights. The module provides a role-based access control mechanism to ensure that different users only view business and technical data related to their responsibilities, ensuring data security.

[0215] ④Data association analysis:

[0216] This module can automatically associate business data with technical data. For example, users can analyze the efficiency of order processing by viewing the system performance data of a certain order (such as response time, server load, etc.), and further associate business information such as order amount and user type for comprehensive analysis.

[0217] Example scenario: An e-commerce company can use this module to view detailed technical and business data of an order processing, including processing time, user experience, payment success rate, etc., and then evaluate the bottlenecks in the business process.

[0218] ⑤Historical data comparison and trend analysis:

[0219] The module supports historical data comparison. Users can select data from multiple time periods for comparison, helping them identify long-term trends and anomalies in systems and business processes. Users can also set thresholds. When system performance or key business indicators exceed or fall below the set range, the system will automatically generate warnings or suggestions.

[0220] Trend analysis function: By analyzing historical data, the module can generate trend charts of business or system performance to help users identify patterns of system performance changes and predict possible problems in the future.

[0221] ⑥Report generation function:

[0222] The module provides a one-click report generation function, which allows users to generate standardized PDF or Excel reports based on the analysis results. These reports can be used as a reference for daily business or technical team meetings to support decision making and problem review.

[0223] Intelligent recommendations and optimization tips:

[0224] The module also integrates intelligent algorithms to generate optimization suggestions by analyzing derived data. For example, if the system detects that operation delays in a certain business process occur frequently, the module will automatically recommend optimization solutions, such as adjusting server configuration or optimizing business logic.

[0225] Example: In an online payment system, if the module detects that VIP users have a low payment success rate, it can suggest optimizing the performance of the payment gateway, or adjusting the payment process priority for VIP customers.

[0226] The derivative data display module is a powerful tool that integrates multiple functions such as high-dimensional data display, interactive analysis, historical data comparison, and intelligent recommendation. It can help enterprises analyze system operation from both business and technical perspectives, quickly locate problems and provide optimization suggestions, thereby effectively supporting business decisions and improving overall system performance.

[0227] (2) Pragmatic data display module

[0228] The pragmatic data display module displays pragmatic data in natural language through prompt engineering. The detailed description is as follows:

[0229] ① Visualization of pragmatic data and natural language generation:

[0230] The module uses multi-dimensional visualization technology and business data, combined with natural language descriptions generated by a large language model, to enable business personnel to easily understand the system operation status and business processes.

[0231] Display format: including technical indicator charts such as line charts and bar charts, as well as business process descriptions generated by natural language. The natural language generation part explains each step in the business process, allowing the technical and business teams to align their understanding.

[0232] Example: In a payment processing scenario, the module can generate a natural language description of "payment confirmation successful, order number 123456, total processing time 12 seconds" and provide a chart showing the processing time.

[0233] ②Interactive analysis and real-time updates:

[0234] The module supports users to filter, sort and drill-down data through an interactive interface, and can update the analysis results in real time. Business personnel can view the latest business process status of the system and further analyze the details of specific business or technical data.

[0235] Real-time: Backend data will be automatically updated to the interface, ensuring that business personnel can view the latest monitoring data at any time. Users can analyze trends and anomalies by filtering specific time periods or business processes.

[0236] Interactive function: allows users to select different dimensions of data for interactive analysis, such as filtering by order amount and viewing the processing time distribution.

[0237] ③ Prediction analysis and optimization suggestions:

[0238] Based on the results of pragmatic data analysis, the module generates predictive analysis of business processes and provides business optimization suggestions. By identifying bottlenecks or anomalies in business processes, the system can automatically recommend adjustment plans, such as optimizing technical configuration or adjusting business strategies.

[0239] Example: In an order processing system, if the order processing time for a certain type of customer is too long, the module will generate optimization suggestions, such as "increase the payment gateway priority of VIP customers to reduce delays."

[0240] ④Historical data comparison and trend display:

[0241] The module allows users to compare historical data from different time periods to help identify long-term trends in business processes. Users can view past analysis results and compare them with current data to evaluate system performance or business process improvements.

[0242] Example: Users can view order processing data for the past three months and use trend charts to find changes in system response speed within a certain period of time, thereby adjusting business strategies or technical configurations.

[0243] The pragmatic data display module combines natural language generation with multi-dimensional visual display to achieve seamless communication between business and technical personnel, breaking the understanding barriers between technology and business in traditional systems, so that business personnel can understand complex system operations and business processes through intuitive natural language interpretation and visual interfaces without in-depth technical background. At the same time, the module provides real-time updates, interactive analysis and optimization suggestions, making business decisions more intelligent and efficient, and solving the problem that traditional systems cannot dynamically process and optimize business processes.

[0244] In summary, the observability system based on business context enhancement proposed in this embodiment realizes richer data collection, processing and display without affecting the functions of the native observable system. Enterprises can not only maintain the reliability and performance monitoring capabilities of existing systems, but also deeply integrate technical data with business processes with enhanced functions to improve the efficiency of risk control and business response.

[0245] The above description of the embodiments is to facilitate the understanding and use of the invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the scope of protection of the present invention.

Claims

1. An observability system based on business context enhancement, built on the observability architecture, including a data collection layer, a data processing layer, a data storage layer and a data display layer, characterized in that: The data collection layer is used to collect metadata of business requirements and technical requirements, and generate technical monitoring data in the business context accordingly; The data processing layer is used to associate the enhanced data with the technical monitoring data in the business context to generate derived data with a unified structure, wherein the enhanced data includes external business data and software life cycle data; the data processing layer is also used to perform semantic analysis on the derived data to generate pragmatic data, wherein the pragmatic data includes business-level operation steps and business optimization suggestions in natural language form; The data storage layer is used to store the derived data and the pragmatic data; the data display layer is used to display the derived data and the pragmatic data in a visual form.

2. The observability system based on business context enhancement according to claim 1, characterized in that: The data collection layer includes a business context management module, a business context injection module and a dynamic configuration management module. The business context management module is used to obtain metadata of business requirements and technical requirements, and establish a business data dictionary based on this. The business data dictionary includes Span information, Span attributes and log information under the business context; the business context injection module integrates an IDE plug-in to inject the information in the business data dictionary into the tracking unit and log of the observability architecture in a structured manner to obtain technical monitoring data under the business context; the dynamic configuration management module is used to dynamically adjust the data collection strategy of the business context management module through a dual-mode strategy.

3. The observability system based on business context enhancement according to claim 2, characterized in that: The process of establishing the business data dictionary is as follows: Extract the business process name from the metadata of the business requirement as the name of the tracking unit Span, and establish a Span information table, wherein the Span information table includes the business process name, business meaning, and business function; Based on the metadata of business requirements and technical requirements, a Span attribute table is established, wherein the Span attribute table includes field information, life cycle status and association identifier of the business entity; The associated actions between entities are extracted from the metadata of business requirements and technical requirements, and a log information table is established. The log information table includes a log number, an associated action name and a log template. The log template includes an associated entity, a behavior type, a value change and additional information.

4. The observability system based on business context enhancement according to claim 2, characterized in that: The process of loading and applying the data collection strategy of the business context management module through the CI / CD component and dynamically adjusting the data collection strategy of the business context management module through the dual-mode strategy by the dynamic configuration management module is as follows: Dynamically adjust the sampling rate, filtering rules, and data buffer size of data collection through the configuration interface provided by the observability SDK; Whether data needs to be collected is determined by an observability function defined based on context information and predefined rules. The context information includes the current business process, user type, and system load.

5. The observability system based on business context enhancement according to claim 1, characterized in that: The data processing layer includes a derivative data offline processing module and an intelligent pragmatic analysis processing module, wherein the derivative data offline processing module is used to obtain enhanced data and associate it with the pre-processed technical monitoring data in the business context to generate derivative data; The intelligent pragmatic analysis processing module is used to perform semantic analysis on the derived data through a large language model to generate pragmatic data.

6. The observability system based on business context enhancement according to claim 5, characterized in that: The specific process of generating the derivative data is as follows: Extracting the business entity ID from the preprocessed technical monitoring data in the business context, obtaining the external business data based on the business entity ID, and associating the preprocessed technical monitoring data in the business context with the external business data through primary key matching, foreign key reference, or consistent hashing; Extract key fields from the preprocessed technical monitoring data in the business context, and obtain software lifecycle data based on the release package ID in the key fields, wherein the software lifecycle data includes submission hash, configuration file version, and build environment information; associate the preprocessed technical monitoring data in the business context with the software lifecycle data through hash matching, mapping table, or index mechanism; According to the hierarchical relationship, association relationship and query requirements of the data, the associated data is uniformly modeled, and the big data processing framework and distributed search engine are used to generate indexes to finally obtain derived data.

7. The observability system based on business context enhancement according to claim 1, characterized in that: The data storage layer includes a derivative data storage module and a pragmatic data storage module, which are respectively used to receive and store the derivative data and the pragmatic data; The derived data storage module first determines the target storage backend of the derived data according to the routing rules, and then further determines the transmission protocol and interface according to the target storage backend, and transmits the data in a parallel transmission mode; The pragmatic data storage module first collects the interaction data of pragmatic data, associates it with corresponding metadata, and establishes pragmatic data records; then designs a data storage structure to store the pragmatic data records.

8. The observability system based on business context enhancement according to claim 7, characterized in that: The derivative data and the pragmatic data are compressed and selectively encrypted before transmission; during the transmission process, data integrity and data legality verification are performed; If the transmission fails, an error log is recorded and a retry is automatically performed.

9. The observability system based on business context enhancement according to claim 7, characterized in that: If the big language model that generates a certain piece of pragmatic data is upgraded, the upgraded big language model is used to generate upgraded pragmatic data based on the interactive data of the pragmatic data, and the upgraded pragmatic data is analyzed for differences with the original pragmatic data to generate optimization suggestions for the big language model.

10. The observability system based on business context enhancement according to claim 1, characterized in that: The data display layer includes a derived data display module and a pragmatic data display module, which are respectively used to display the derived data and the pragmatic data. The pragmatic data display module displays the pragmatic data in a natural language form through a prompting project.

Images