Data production and application method based on index management

Abstract

The invention discloses a data production and application method based on index management, and the method comprises the steps: receiving an index query request containing a target index identifier, a dimension constraint condition and a time range parameter, carrying out the analysis and semantic verification of the target index identifier based on a global index asset library, and obtaining the index definition information; generating a standardized query statement according to the index definition information, and selecting an optimal data calculation engine to execute query; performing multi-level cache optimization and parallel computing acceleration on the query process to obtain an original data set; processing the original data set in real time according to a preset analysis model to generate structured index data containing trend analysis, anomaly detection or attribution inference results; and filtering and desensitizing the data based on a fine-grained permission control strategy, only returning contents in a permission range and recording an audit log. By means of the method, unified management, efficient query and intelligent analysis of the index data are achieved, the automation level and query performance of data production are improved, and meanwhile data safety and access controllability are guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, and in particular to a data production and application method based on index management. BACKGROUND

[0002] In the field of modern enterprise data management and analysis, as the core basis for quantifying business operation and performance, efficient and reliable data supply of indexes is of key significance to business decision-making. At present, the traditional method commonly used in the industry relies on the combination of data warehouse and business intelligence tools. According to business needs, technical teams manually write query scripts to extract data from underlying data tables and generate static reports or export files for use. This method has supported the basic data analysis needs of enterprises for a long period of time, but due to the lack of a unified business semantic layer and automated production mechanism, it leads to long development cycle of indexes, chaotic caliber management, and difficulty in adapting to the growing demand for multi-dimensional and real-time data analysis. Especially when facing large amounts of data and complex business logic, query performance often becomes a bottleneck, making it impossible to support high concurrency and low latency interactive analysis scenarios. In addition, in the process of data sharing and use, the granularity of permission control is coarse, and the audit mechanism is not perfect, which also increases the complexity of data security and management. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a data production and application method based on index management, which realizes semantic unification and automated query generation by building a global index asset library, and solves the problems of inefficient enterprise index data production, chaotic caliber and insufficient query performance by combining intelligent routing and cache acceleration mechanisms.

[0004] In order to achieve the above technical purpose, the technical solution adopted by the present application is as follows: a data production and application method based on index management, comprising: receiving an index query request, the index query request containing a target index identifier, dimension constraint conditions and time range parameters; analyzing and verifying the target index identifier based on a global index asset library to obtain verified index definition information, the global index asset library storing definition information of a plurality of global indexes, each global index including an atomic index, a derived index and a composite index, and the definition information including business caliber, technical calculation logic, associated data sources and dimensions of the index; generating a standardized query statement adapted to the structure of the underlying data warehouse according to the index definition information through an index-data mapping relationship, and selecting the optimal data calculation engine to execute the query according to an intelligent routing strategy; A multi-level cache optimization and parallel computing acceleration are applied to the query execution process to obtain the original data set. The multi-level cache optimization includes preheating storage and dynamic updating of the query results of high-frequency indicators. The parallel computing acceleration adopts an MPP architecture for distributed processing. The original data set is processed in real time according to a preset analysis model to generate structured indicator data with trend analysis, anomaly detection or attribution inference results. The structured indicator data is filtered and desensitized based on a fine-grained permission control strategy, and only the data content accessible within the permission range of the requester is returned to the requester. Complete data access behavior is recorded in an audit log for traceability analysis.

[0005] In some embodiments, the global indicator asset library is generated by the following steps: An indicator registration request is received, which includes the business scope, technical calculation logic, associated data source and dimension information of the indicator to be registered; According to the reuse level of the technical calculation logic, the to-be-registered indicator is classified as an atomic indicator, a derived indicator or a composite indicator; If the to-be-registered indicator is an atomic indicator, it is mapped and modeled with the measurement field and its default aggregation method of the core fact table in the data warehouse; If the to-be-registered indicator is a derived indicator or a composite indicator, it is derived and modeled based on the defined atomic indicators or derived indicators by configuring dimensions, filtering conditions and operation formulas; The mapping relationship between the to-be-registered indicator output after the above modeling and the data warehouse table, field and calculation rule is persisted to form traceable indicator-data bloodline association and form indicator definition information; The indicator definition information is stored in the global indicator asset library, and a globally unique versioned identifier is assigned to the indicator associated with the indicator definition information; The indicator definition information in the global indicator asset library is subjected to lifecycle management, including automatic inspection and archiving based on access frequency and business relevance.

[0006] In some embodiments, the target indicator identifier is parsed and semantically verified based on the global indicator asset library to obtain the verified indicator definition information, including: The complete indicator definition information associated with the identifier is retrieved from the global indicator asset library; The retrieved indicator definition information is subjected to syntax compliance check to verify the integrity and standardization of the business scope, technical calculation logic, associated data source and dimension information of the retrieved indicator definition information; The indicator definition information that passes the syntax check is subjected to semantic consistency verification according to a preset indicator semantic rule library to ensure that there is no contradiction or conflict between its business scope and technical calculation logic; When detecting semantic ambiguity or conflict in the indicator definition information, trigger the indicator caliber abnormality processing procedure, generate a check report containing specific conflict items and correction suggestions; Attach a digital signature and a timestamp to the indicator definition information that passes all checks, and generate checked indicator definition information that can be trusted by subsequent query execution components.

[0007] In some embodiments, a standardized query statement that adapts to the underlying data warehouse structure is generated according to the indicator definition information through the indicator-data mapping relationship, including: Parse the checked indicator definition information, extract technical calculation logic, associated data source identification and dimension information as query generation elements; Determine the corresponding data warehouse table, field name and association based on the query generation elements through the indicator-data mapping relationship; According to the business caliber and technical calculation logic of the indicator definition information, convert the dimension constraint condition and time range parameter into the corresponding SQL filter condition clause; According to the data warehouse hierarchical model optimization principle, automatically select the optimal data level and construct the complete query main body containing the select list, group by grouping and aggregation function; Use the query optimization algorithm based on abstract syntax tree to optimize the generated query statement structure, eliminate redundant calculation and ensure that the syntax conforms to the target computing engine specification, and output the optimized and completely adapted standardized query statement to the underlying data warehouse structure.

[0008] In some embodiments, the optimal data computing engine is selected to execute the query according to the intelligent routing strategy, including: Obtain the query statement features of the standardized query statement, including syntax features, data volume estimation results and complexity scores; Match the pre-configured engine capability profile based on the query statement features, and the engine capability profile contains the performance and resource consumption characteristics of each data computing engine for specific query types; Combine real-time load monitoring data, and perform comprehensive evaluation on query delay, resource utilization and calculation cost through multi-objective optimization algorithm to obtain evaluation results; According to the evaluation results, select the optimal target execution engine from the available engine cluster, and generate the corresponding query task distribution instructions; Establish a session connection between the query task and the execution engine, and complete the initialization configuration of the query context environment.

[0009] In some embodiments, multi-level cache optimization and parallel computing acceleration are applied to the query execution process to obtain the original data set, including: Before query execution, based on the query statement features and historical execution records, cache matching retrieval is performed. If the cache result is hit, the cached data is directly returned; For query requests that do not hit the cache, a cache warming strategy based on a cost model is used to identify high-frequency access patterns and precompute and store intermediate results; Complex query tasks are decomposed into multiple sub-query units that can be executed in parallel, and an optimal parallel execution plan is generated by a query optimizer; A dynamic resource scheduling algorithm is used to allocate appropriate computing resources to each sub-query unit, and intelligent cache management is implemented for intermediate results during execution, establishing a mapping relationship between query statement features and cached data; The final query result is stored in a compressed manner and the cache metadata information is updated, generating the original data set.

[0010] In some embodiments, a dynamic resource scheduling algorithm is used to allocate appropriate computing resources to each sub-query unit, and intelligent cache management is implemented for intermediate results during execution, establishing a mapping relationship between query statement features and cached data, including: Based on the computational complexity characteristics and data dependency graph of the sub-query unit, a multi-objective optimization algorithm is used to allocate resources evenly among computing nodes; The resource allocation strategy is dynamically adjusted according to the actual execution progress of each unit; The value of the intermediate result data generated during execution is evaluated, and the cache priority is determined based on the access frequency prediction model; A mapping index of query feature vectors and cache data blocks is constructed, and a consistent hashing algorithm is used to achieve efficient positioning of distributed cache; And, the cache content is dynamically maintained according to the data access frequency and freshness requirements.

[0011] In some embodiments, the original data set is processed in real time according to a pre-set analysis model to generate structured indicator data with trend analysis, anomaly detection or attribution inference results, including: Data quality verification and missing value processing are performed on the original data set to ensure the integrity and accuracy of the input data; According to the business scope of the indicator definition information, the corresponding analysis model is selected, including a time series prediction model, an isolation forest anomaly detection model, or a SHAP attribution analysis model; The pre-processed original data set is input into the selected analysis model to perform batch or streaming computation, generating preliminary analysis results; The preliminary analysis results are post-processed and optimized, including result smoothing processing, confidence interval calculation, and significance annotation; The optimized preliminary analysis result is associated and integrated with the original index data to generate structured index data containing basic data and analysis conclusion.

[0012] In some embodiments, the structured index data is filtered and desensitized based on a fine-grained permission control policy, including: The identity credential and access context of the requester are parsed, and the permission attribute is extracted, including the user identifier, role permission, and data access range constraint corresponding to the requester; According to the extracted permission attribute, a set of predefined access control rules is matched to determine the access range of the requester to the target index data; Based on the determined access range, row and column level data filtering is implemented on the structured index data to remove data content outside the permission range of the requester; A dynamic desensitization algorithm is applied to sensitive fields in the retained data, and differential desensitization processing is implemented according to the field sensitivity level and user permission; An audit log record containing data filtering and desensitization processing details is generated, and an association relationship with the processed index data is established to construct a standardized data response package.

[0013] In some embodiments, the requester is returned with the data content accessible within its permission range, and the complete data access behavior is recorded in the audit log for traceability analysis, including: The standardized data response package is returned to the requester through a secure transmission channel, and data fingerprint verification information is added in the response header of the standardized data response package; Real-time capture of full-link behavior information for this data access, including requester identity, access time, query condition, returned data volume, and processing time consumption; The captured full-link behavior information is associated and integrated with the corresponding index definition information and permission control policy to generate a structured audit log record; The audit log is encrypted and stored using blockchain storage technology.

[0014] Compared with the prior art, the method has the beneficial effects that: by receiving an index query request containing a target index identifier, a dimension constraint condition and a time range parameter, the target index identifier is parsed and semantically verified based on a global index asset library to obtain index definition information; a standardized query statement is generated according to the index definition information, and an optimal data calculation engine is selected to execute the query; multi-level cache optimization and parallel computing acceleration are implemented on the query process to obtain an original data set; the original data set is processed in real time according to a preset analysis model to generate structured index data containing trend analysis, anomaly detection or attribution inference results; and the data is filtered and desensitized based on a fine-grained permission control strategy, and only the content within the permission range is returned and an audit log is recorded. The method realizes unified management, efficient query and intelligent analysis of index data, improves the automation level of data production and query performance, and guarantees data security and access controllability. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0016] Figure 1 is a method step diagram of steps S101 to S106 of the method described in the specific embodiment; Figure 2 is a method step diagram of steps S201 to S206 of the method described in the specific embodiment. DETAILED DESCRIPTION

[0017] The present application will be further described in detail below in combination with the drawings and embodiments. It is particularly pointed out that the following embodiments are only used to illustrate the present application, but do not limit the scope of the present application. Similarly, the following embodiments are only some embodiments of the present application, not all embodiments, and all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application.

[0018] Please refer to Figure 1 The present embodiment provides a data production and application method based on index management, comprising: S101, receiving an index query request, the index query request containing a target index identifier, a dimension constraint condition and a time range parameter; S102, based on the global index asset library, the target index identifier is parsed and semantically verified, and the verified index definition information is obtained. The global index asset library stores the definition information of a plurality of global indexes. Each global index includes an atomic index, a derived index and a composite index. The definition information includes the business scope, the technical calculation logic, the associated data source and the dimension of the index. S103, according to the index definition information, a standardized query statement adapted to the underlying data warehouse structure is generated through the index-data mapping relationship, and the optimal data calculation engine is selected according to the intelligent routing strategy to execute the query; S104, multi-level cache optimization and parallel computing acceleration are applied to the query execution process to obtain the original data set. The multi-level cache optimization includes preheating storage and dynamic updating of high-frequency index query results. The parallel computing acceleration adopts MPP architecture for distributed processing. S105, the original data set is processed in real time according to the preset analysis model to generate structured index data with trend analysis, anomaly detection or attribution inference results; S106, based on the fine-grained permission control strategy, the structured index data is filtered and desensitized, only the data content accessible within the permission range of the requester is returned, and the complete data access behavior is recorded in the audit log for traceability analysis.

[0019] In step S101, the index query request is a standardized data acquisition instruction initiated by a business user. The target index identifier is used to uniquely determine the index object to be queried. The dimension constraint condition is used to limit the grouping and filtering range of the data. The time range parameter is used to specify the time interval of the data. This step receives the request through a unified interface to provide input for subsequent automated processing.

[0020] In step S102, the global index asset library is a core component that centrally stores and manages all index definitions. The atomic index is an indivisible basic measurement. The derived index is derived from the atomic index through dimension constraint or simple operation. The composite index is obtained by composite calculation of multiple indexes. The business scope describes the business meaning and rules of the index. The technical calculation logic defines the specific calculation method of the index. The associated data source and dimension indicate the data table and dimension attribute on which the index depends. The parsing and semantic verification process ensures the integrity and consistency of the index definition through a verification algorithm to avoid ambiguity.

[0021] In step S103, the index-data mapping relationship establishes the association between the index definition and the underlying data warehouse table, field and calculation rule, which is used to automatically generate a standardized query statement. The intelligent routing strategy dynamically selects the optimal data calculation engine (such as Presto or Spark) based on the query complexity, data volume and engine load state to improve the query efficiency and resource utilization.

[0022] In step S104, the multi-level cache optimization reduces repeated calculations by identifying high-frequency query patterns and preheating and dynamically updating query results. Parallel computing acceleration uses an MPP architecture to divide query tasks into multiple subtasks for parallel execution, significantly improving the processing speed of large data queries.

[0023] In step S105, the preset analysis model includes standardized analysis algorithms such as trend analysis, anomaly detection, and attribution inference, which are used to process the original data set in real time and generate structured index data with analysis conclusions, enhancing the business insight of the data.

[0024] In step S106, the fine-grained permission control strategy implements row and column level filtering and dynamic desensitization on structured index data based on user roles and data sensitivity levels, ensuring data security. Audit logs record complete data access behavior, supporting operation traceability and compliance auditing.

[0025] The embodiment receives an index query request, analyzes and verifies based on the global index asset library, automatically generates a standardized query statement and selects the optimal computing engine for execution, combines cache optimization and parallel computing acceleration to obtain data, and returns the results according to the permission control after real-time analysis and processing. This method realizes unified index caliber, efficient and automatic query, and controllable data security through full-link automatic processing, effectively improving data production efficiency and quality.

[0026] Please refer to Figure 2 In some embodiments, the global index asset library is generated by the following steps: S201, receiving an index registration request, the index registration request containing the business caliber, technical calculation logic, associated data source and dimension information of the to-be-registered index; S202, according to the reuse level of the technical calculation logic, classifying the to-be-registered index into atomic index, derived index or composite index; S203, if the to-be-registered index is an atomic index, mapping it with the measurement field of the core fact table in the data warehouse and its default aggregation method for modeling; If the to-be-registered index is a derived index or a composite index, based on the defined atomic index or derived index, derive modeling by configuring dimensions, filtering conditions and operation formulas; S204, persisting the mapping relationship between the to-be-registered index output after the above modeling and the data warehouse table, field and calculation rule, forming traceable index-data blood relationship, forming index definition information; S205, storing the index definition information into the global index asset library, and assigning a globally unique versioned identifier to the index associated with the index definition information; S206, the life cycle management is performed on the index definition information in the global index asset library, including automatic inspection and archiving based on access frequency and business correlation.

[0027] In step S201, the index registration request is a standardized index creation application submitted by an index definition personnel, containing core metadata such as business scope, technical calculation logic, associated data source and dimension information, wherein the business scope is used to describe the business meaning and statistical rules of the index, the technical calculation logic defines the specific calculation expression and data processing flow of the index, and the associated data source and dimension information specifies the data table and dimension attribute on which the index depends. This step ensures the standardization and completeness of the index definition.

[0028] In step S202, the reuse level is determined according to the dependency relationship and complexity of the technical calculation logic: the atomic index is an indivisible basic measurement, directly derived from the measurement field of the core fact table of the data warehouse; the derived index is derived from the atomic index through dimension constraint, filtering condition or simple operation; the composite index is obtained by composite calculation of multiple atomic indexes or derived indexes. The classification process is automatically completed through a rule engine, ensuring the clear hierarchy and reusability of the index system.

[0029] In step S203, mapping modeling establishes a direct association with the measurement field of the core fact table and its default aggregation method (such as SUM, COUNT) for atomic indexes; for derived indexes and composite indexes, based on the defined atomic indexes or derived indexes, the dimensions, filtering conditions and operation formulas are configured to realize the flexible combination of index logic. Preferably, a visual configuration interface is used to support business personnel to participate in index definition, reducing the technical threshold.

[0030] In step S204, the mapping relationship persistence stores the association information of the index with the data warehouse table, field and calculation rule in the metadata management system, forming a traceable index-data blood relationship. The blood relationship supports tracing the data source of the index upwards and tracking the use of the data downwards, enhancing the transparency and credibility of the data.

[0031] In step S205, a globally unique versioned identifier is assigned through an identification generation algorithm, ensuring the uniqueness and version traceability of the index definition information. The versioned identifier supports iterative updating and historical version management of the index definition, avoiding data inconsistency problems caused by index changes.

[0032] In step S206, the life cycle management sets automatic inspection rules based on access frequency, business correlation and other dimensions, identifies low-usage indexes and triggers the archiving process, ensuring the simplicity and efficiency of the index asset library. Automatic inspection is performed through a timing task, reducing the cost of manual maintenance.

[0033] The embodiment builds a unified and traceable global index asset library through standardized index registration, classification modeling and blood management, provides basic support for the consistency management and efficient reuse of indexes, and effectively solves the problems of index caliber confusion and repeated development.

[0034] In some embodiments, based on the global index asset library, the target index identifier is parsed and semantically verified to obtain verified index definition information, including: retrieve complete index definition information associated with the identifier from the global index asset library; perform syntax compliance checking on the retrieved index definition information to verify the completeness and standardization of the business caliber, technical calculation logic, associated data source and dimension information of the retrieved index definition information; perform semantic consistency verification on the index definition information that passes the syntax check according to the preset index semantic rule library to ensure that there is no contradiction between the business caliber and the technical calculation logic; When semantic ambiguity or conflict is detected in the index definition information, trigger the index caliber exception handling process to generate a verification report containing specific conflict items and correction suggestions; attach a digital signature and a timestamp to the index definition information that passes all verifications to generate verified index definition information that can be trusted by subsequent query execution components.

[0035] In the embodiment, retrieving complete index definition information associated with the identifier from the global index asset library can be understood as performing an exact match query in the asset library through the index identifier to obtain complete metadata including business caliber, technical calculation logic, associated data source and dimension information, etc. This step ensures that subsequent processing is based on accurate index definition.

[0036] Syntax compliance checking verifies the completeness and standardization of the index definition information. Business caliber checking ensures that the business description is unambiguous and complies with the naming specification. Technical calculation logic verification ensures that the calculation expression syntax is correct and executable. Associated data source and dimension information checking confirms that the data table structure and dimension attributes exist and are accessible. This check is automatically performed through a predefined syntax rule set to prevent calculation errors caused by incomplete definitions.

[0037] The index semantic rule library stores the mapping relationship between business rules and technical rules. Semantic consistency verification compares the matching degree of business caliber description and technical calculation logic through a rule engine to ensure that there is no contradiction between the two. For example, verify whether the business caliber of "order amount" is consistent with the SUM aggregation logic in technical calculation to avoid semantic ambiguity.

[0038] When semantic ambiguity or conflict is detected, the index caliber abnormality processing flow is automatically triggered, the specific conflict item is located through a pattern recognition algorithm, and a check report containing correction suggestions is generated based on a rule base. Preferably, the flow supports manual intervention for review to ensure the accuracy of the abnormality processing.

[0039] The digital signature and timestamp attach a trusted identification to the verified index definition information through an asymmetric encryption algorithm and a time service. The digital signature ensures information integrity and source credibility, and the timestamp records the verification time point. This enables the subsequent query execution component to directly trust and use the index definition information without repeated verification.

[0040] The present embodiment ensures the consistency of index definition at the syntax and semantic levels through a multi-level verification mechanism, and establishes a trusted transmission mechanism in combination with a digital signature technology, thereby fundamentally guaranteeing the accuracy and reliability of index data and effectively solving the problem of inconsistent caliber across departments.

[0041] In some embodiments, a standardized query statement adapted to the underlying data warehouse structure is generated according to the index definition information through an index-data mapping relationship, including: The parsed and verified index definition information is extracted to obtain technical calculation logic, associated data source identification, and dimension information as query generation elements; Based on the query generation elements, the corresponding data warehouse table, field name, and association are determined through the index-data mapping relationship; According to the business caliber and technical calculation logic of the index definition information, the dimension constraint condition and time range parameter are converted into corresponding SQL filter condition clauses; According to the data warehouse hierarchical model optimization principle, the optimal data level is automatically selected and a complete query main body including a select list, a group by grouping, and an aggregation function is constructed; An abstract syntax tree-based query optimization algorithm is used to optimize the generated query statement in structure, eliminate redundant calculations, and ensure that the syntax conforms to the target computing engine specification, thereby outputting an optimized and completely adapted standardized query statement to the underlying data warehouse structure.

[0042] In the present embodiment, the semantically verified index definition is structurally parsed to extract technical calculation logic, associated data source identification, and dimension information as query generation elements. The technical calculation logic includes the calculation expression and aggregation method of the index, the associated data source identification indicates the physical table where the data is located, and the dimension information determines the grouping and filtering dimensions of the data.

[0043] The corresponding data warehouse table, field name and association are determined by index-data mapping relationship based on query generation elements, which means that the logical index elements are converted into physical data objects by using the pre-defined mapping rules. The mapping relationship maintains the correspondence between the index and the underlying data table field, ensuring that the query can be correctly associated to the actual data storage location.

[0044] The dimension constraint condition and time range parameter are converted into corresponding SQL filter condition clauses, and the business level filtering condition is converted into a database executable WHERE clause through a condition parser. The dimension constraint condition is converted into an equal or range filter, and the time range parameter is converted into an interval condition of the time field, ensuring that the query result meets the business filtering requirements.

[0045] The optimal data level is automatically selected according to the data warehouse hierarchical model optimization principle, and the data is intelligently selected from the DWD detail layer, DWS summary layer or ADS application layer according to the query complexity, data volume and performance requirements. The selection is based on the comprehensive evaluation of data freshness, query performance and data volume, and the complete query main body including select list, group by grouping and aggregation function is constructed.

[0046] An abstract syntax tree-based query optimization algorithm is used to optimize the generated query statement structure, which parses the syntax tree structure of the SQL statement, eliminates redundant calculations, optimizes join order and predicate pushdown, and ensures that the syntax conforms to the target computing engine specification. The optimization process outputs a standardized query statement that fully adapts to the underlying data warehouse structure, improving query execution efficiency.

[0047] This embodiment converts business index requirements into efficient standard SQL queries through an automated query statement generation and optimization mechanism, realizing seamless conversion from business logic to data query, effectively solving the problem of low efficiency and error-prone of manual SQL writing.

[0048] In some embodiments, the optimal data computing engine is selected to execute the query according to the intelligent routing strategy, including: Obtain the query statement features of the standardized query statement, including syntax features, data volume estimation results and complexity scores; Match the pre-configured engine capability profile based on the query statement features, and the engine capability profile contains the performance and resource consumption characteristics of each data computing engine for specific query types; Combine real-time load monitoring data and perform comprehensive evaluation on query delay, resource utilization and computing cost through a multi-objective optimization algorithm to obtain an evaluation result; Select the optimal target execution engine from the available engine cluster according to the evaluation result, and generate corresponding query task distribution instructions; A session connection between the query task and the execution engine is established, and initialization configuration of a query context environment is completed.

[0049] In the embodiment, the query statement features refer to a feature set obtained by performing syntax analysis and statistical analysis on the standardized query statement, wherein the syntax features reflect the structural characteristics of the query statement, the data volume estimation result estimates the data scale involved in the query based on statistical information, and the complexity score comprehensively evaluates the computational complexity of the query.

[0050] The engine capability profile is a capability description of each data computing engine constructed through historical performance data collection and analysis, and includes performance and resource consumption characteristics of specific query types. The performance includes query response time and throughput, and the resource consumption characteristics include CPU, memory, and IO usage patterns. Preferably, the engine capability profile is continuously updated and optimized by machine learning methods from historical execution records.

[0051] The multi-objective optimization algorithm is a mathematical optimization method for balancing multiple objectives such as query delay, resource utilization, and computing cost. By establishing objective functions and constraint conditions, and combining real-time load monitoring data, a comprehensive evaluation result is calculated. The algorithm uses weighted summation or Pareto optimization strategies to ensure that the selection result is balanced in multiple performance dimensions.

[0052] The engine scheduling instruction generated according to the evaluation result includes target execution engine identification, query statement, and execution parameters. The query task distribution instruction is sent to the target engine through a message queue or API call to ensure accurate distribution of the query task to the optimal execution node.

[0053] The session connection is a communication link established between the query task and the execution engine for transmitting query requests and returning results. The initialization configuration of the query context environment includes setting connection parameters, loading necessary data dictionaries, and pre-allocating computing resources, etc. to ensure that the query can be executed in the optimal environment.

[0054] The embodiment realizes the best matching of the query task and the computing engine through an intelligent engine selection mechanism, fully utilizes the strengths of different engines, optimizes resource utilization while ensuring query performance, and effectively improves the query efficiency and resource utilization of the overall system.

[0055] In some embodiments, multi-level cache optimization and parallel computing acceleration are applied to the query execution process to obtain the original data set, including: Before query execution, cache matching retrieval is performed based on query statement features and historical execution records, and if the result cache is hit, the cache data is directly returned; For query requests that do not hit the cache, a cache warm-up strategy based on a cost model is used to identify high-frequency access patterns and precompute and store the corresponding intermediate results. The complex query task is divided into multiple sub-query units that can be executed in parallel, and an optimal parallel execution plan is generated by a query optimizer. An appropriate amount of computing resources is allocated to each sub-query unit using a dynamic resource scheduling algorithm, intelligent cache management is implemented for intermediate results during execution, and a mapping relationship between query statement features and cached data is established. The final query result is stored in a compressed format and the cache metadata information is updated to generate the original data set.

[0056] In this embodiment, cache matching retrieval is a process of determining whether the current query can reuse existing cached data by comparing and analyzing query statement features and historical execution records. Query statement features include syntax structure, filter conditions, and aggregation patterns, and historical execution records store cache hit information for past queries. This retrieval achieves fast matching through feature similarity calculation, and if the cache hits the result, the cached data is returned directly to avoid repeated calculation.

[0057] The cache warm-up strategy based on the cost model refers to intelligently predicting and precomputing intermediate results corresponding to high-frequency access patterns according to factors such as query frequency, data update period, and computing cost. This strategy identifies hot data by analyzing query history patterns and precomputes and stores them during low-load periods of the system, effectively improving cache hit rate and query response speed.

[0058] The query optimizer determines the optimal parallelism based on data distribution characteristics and computing resource status through cost evaluation to ensure efficient collaborative execution of each sub-query unit. The dynamic resource scheduling algorithm allocates appropriate computing resources in real time based on the computing complexity and priority of each sub-query unit. This algorithm monitors system resource utilization, intelligently adjusts CPU, memory, and other resource allocation, and implements intelligent cache management for intermediate results during execution to establish a mapping relationship between query statement features and cached data, achieving flexible allocation and efficient utilization of resources.

[0059] Columnar compression or dictionary encoding algorithms are used to reduce storage space for the final query result, and cache metadata information is updated to record attributes such as data storage location, compression method, and validity period. This process generates a structured original data set, providing an efficient storage foundation for subsequent data applications.

[0060] This embodiment achieves significant improvement in query performance through the organic combination of multi-level cache optimization and parallel computing acceleration, reducing resource consumption caused by repeated calculations and shortening query response time through parallel processing, effectively solving the performance bottleneck problem of large data volume queries.

[0061] In some embodiments, a dynamic resource scheduling algorithm is adopted to allocate appropriate computing resources to each subquery unit, intelligent cache management is implemented for intermediate results during execution, and a mapping relationship between query statement features and cached data is established, including: Based on the computational complexity features and data dependency graph of the subquery units, a multi-objective optimization algorithm is used to allocate resources evenly among the computing nodes; The resource allocation strategy is dynamically adjusted according to the actual execution progress of each unit; The value of the intermediate result data generated during execution is evaluated, and the cache priority is determined based on the access frequency prediction model; A mapping index of query feature vectors and cache data blocks is constructed, and a consistent hashing algorithm is used to achieve efficient positioning of distributed caches; And the cache content is dynamically maintained according to the data access frequency and freshness requirements.

[0062] In this embodiment, the computational complexity features are quantitative evaluations of the computational intensity, data size, and operation type of the subquery units, and the data dependency graph describes the data flow and dependency relationship between the subquery units. The multi-objective optimization algorithm allocates resources evenly among the computing nodes based on these features, taking into account multiple optimization objectives such as computational load balancing, data transmission cost, and execution efficiency, to ensure the optimality of resource allocation.

[0063] Dynamic adjustment of resource allocation strategy is to adjust the allocation scheme of CPU, memory and other computing resources in real time according to the actual execution progress and system resource state of each unit. This strategy monitors execution indicators and resource utilization, dynamically optimizes resource allocation through a feedback control mechanism, and ensures the overall efficiency and stability of query execution.

[0064] Value evaluation is an importance evaluation of the intermediate result data generated during execution, based on the access frequency prediction model to analyze the probability of repeated use of data. This model determines the cache priority through historical access patterns and query feature analysis, preferentially caches high-value intermediate result data, and improves cache utilization efficiency.

[0065] Query feature vector is a numerical vector quantifying the structural features, filter conditions, and aggregation operations of the query statement, and cache data block is a storage unit of intermediate results. Building a mapping index can be understood as establishing an association between queries and cached data through feature matching, and using a consistent hashing algorithm to achieve efficient positioning and load balancing of cached data in a distributed environment.

[0066] According to the data access frequency and freshness requirements, the cache data is automatically updated and eliminated, and the cache content is dynamically maintained. Data with high access frequency is preferentially retained, and data with expired or low access frequency is timely cleaned up, ensuring efficient use of cache space and data consistency.

[0067] The embodiment realizes optimal configuration of computing resources and cache resources through intelligent resource scheduling and cache management mechanism, guarantees high efficiency of query execution, and improves data reuse rate through intelligent cache, effectively solving the key problems of resource utilization and performance optimization in a distributed environment.

[0068] In some embodiments, the original data set is processed in real time according to a preset analysis model to generate structured index data with trend analysis, anomaly detection or attribution inference results, including: Data quality verification and missing value processing are performed on the original data set to ensure the integrity and accuracy of the input data; According to the business scope of the index definition information, the corresponding analysis model is selected, including a time series prediction model, an isolated forest anomaly detection model or a SHAP attribution analysis model; The preprocessed original data set is input into the selected analysis model to perform batch or streaming calculation, generating preliminary analysis results; The preliminary analysis results are post-processed and optimized, including result smoothing processing, confidence interval calculation and significance annotation; The optimized preliminary analysis results are associated and integrated with the original index data to generate structured index data containing basic data and analysis conclusions.

[0069] In the embodiment, data quality verification is a process of verifying the integrity, consistency and accuracy of the original data set, and missing value processing includes identifying and filling data gaps. These preprocessing operations are realized through data exploration and statistical analysis methods to ensure the integrity and accuracy of the input data and provide a reliable data foundation for subsequent analysis.

[0070] The business scope of the index definition information refers to the definition specification and calculation rules of the index at the business level, and the selection of the analysis model is based on the matching of these business requirements. The time series prediction model is used for trend analysis, the isolated forest anomaly detection model identifies data anomalies, and the SHAP attribution analysis model explains the reasons for index fluctuations. The selection of these models ensures that the analysis results are highly consistent with business requirements.

[0071] Batch or stream computing refers to selecting different computing modes according to data size and timeliness requirements. Batch computing is suitable for historical data analysis, and stream computing supports real-time data processing. The preprocessed original data set is input into the selected analysis model to perform calculation, generating preliminary analysis results including trend prediction, anomaly marking or attribution analysis.

[0072] Post-processing optimization refers to further processing of preliminary analysis results. Result smoothing eliminates random fluctuations, confidence interval calculation evaluates result reliability, and significance annotation identifies important change points. These optimization operations are achieved through statistical methods and algorithm processing, improving the readability and practicality of analysis results.

[0073] Structural association of optimized preliminary analysis results with original index data generates complete output containing basic data and analysis conclusions. Association and integration are achieved through data association technology and structured storage, forming structured index data that can be directly used for business decision-making.

[0074] This embodiment realizes the automatic conversion from raw data to intelligent analysis results through standardized analysis model application and systematic post-processing process, ensuring the standardization of the analysis process and enhancing the business value of data through multi-dimensional analysis, effectively improving the analysis depth and practical value of index data.

[0075] In some embodiments, the structured index data is filtered and desensitized based on a fine-grained permission control strategy, including: Parse the identity credentials and access context of the requestor, extract the permission attributes, including the user identifier, role permission and data access range constraint corresponding to the requestor; According to the extracted permission attributes, match the pre-defined access control rule set to determine the access range of the target index data for the requestor; Based on the determined access range, implement row and column level data filtering on the structured index data, remove data content outside the requestor's permission range; Apply dynamic desensitization algorithm to sensitive fields in the retained data, implement differential desensitization processing according to field sensitivity level and user permission; Generate audit log records containing data filtering and desensitization processing details, and establish an association relationship with the processed index data to build a standardized data response package.

[0076] In this embodiment, the identity credentials and access context are authentication information and access environment parameters provided by the requestor, and the permission attributes include user identifier, role permission and data access range constraint, etc. key attributes. By analyzing these information, the system can accurately identify the identity characteristics and permission boundaries of the requestor, providing decision basis for subsequent access control.

[0077] The access control rule set refers to a predefined set of data access permission rules, which are matched based on the extracted permission attributes to determine the accessible range of the target indicator data for the requestor. This matching is implemented through a rule engine to ensure the accuracy and consistency of the permission determination and effectively control the data access boundary.

[0078] Row-column level data filtering refers to fine-grained filtering of structured indicator data based on the determined access range to remove data rows and columns outside the requestor's permission range. This filtering is implemented through SQL condition screening or data processing to ensure that users can only access data within the authorized range, achieving data security isolation.

[0079] The dynamic desensitization algorithm implements differentiated data desensitization processing according to the field sensitivity level and user permissions, and performs content transformation on sensitive fields in the retained data. This algorithm uses techniques such as masking, hashing, or encryption to protect sensitive information while ensuring data usability, achieving a balance between data security and usage needs.

[0080] Audit log recording refers to operation records containing data filtering and desensitization processing details, and establishes an association with the processed indicator data. Audit log recording is automatically generated through a log system to build a standardized data response package, providing complete evidence for data security audit and problem tracing.

[0081] This embodiment realizes fine-grained security management of structured indicator data through multi-level permission control and intelligent data processing mechanisms, ensuring data access compliance and ensuring data usage traceability through perfect audit tracking, effectively solving the contradiction between data security and sharing utilization.

[0082] In some embodiments, the requestor is returned with data content within their permission range, and the complete data access behavior is recorded in the audit log for traceability analysis, including: Return the standardized data response package to the requestor through a secure transmission channel, and add data fingerprint verification information in the response header of the standardized data response package; Real-time capture of full-link behavior information for this data access, including requestor identity, access time, query conditions, returned data volume, and processing time; Integrate the captured full-link behavior information with corresponding indicator definition information and permission control strategies to generate structured audit log records; Use blockchain storage technology to encrypt and store the audit log.

[0083] In this embodiment, the secure transmission channel is a data transmission link established by using an encryption protocol such as TLS / SSL, which is used to securely return the standardized data response package to the requester. The data fingerprint verification information is a unique identifier generated by a hash algorithm and added in the response header to verify data integrity and prevent tampering, ensuring the security and reliability of the data transmission process.

[0084] The full-link behavior information refers to the complete operation record generated during data access, including the requester's identity, access time, query conditions, returned data volume, and processing time consumption, and other key information. By capturing these information in real time, the system can comprehensively monitor data access behavior and provide complete raw data basis for audit analysis.

[0085] The captured full-link behavior information is logically associated with the corresponding index definition information and permission control strategy to generate structured audit log records. Through data association technology and structured storage, the associated integration is realized to form audit records containing complete context information, facilitating subsequent traceability analysis.

[0086] The blockchain storage technology uses distributed ledger technology to encrypt and store audit logs, ensuring the non-tamperability and traceability of log records through hash chain structure and distributed consensus mechanism. The blockchain storage technology realizes trusted audit evidence for data access behavior through encryption algorithms and distributed storage.

[0087] This embodiment realizes controllable and traceable data access through the combination of secure transmission, complete recording, and trusted storage technology, ensuring the security of data transmission and the non-tamperability of audit records through blockchain technology, providing reliable technical support for data security governance.

[0088] By adopting the above technical solutions, the present application is different from the prior art and has the following beneficial effects: by constructing a global index asset library to realize unified standardized management of index caliber, automatically generating standardized query statements based on index-data mapping relationship, selecting the optimal calculation engine to execute queries combined with intelligent routing strategy, and using multi-level cache optimization and parallel computing acceleration technology to improve query efficiency. By presetting analysis models to process raw data in real time to generate structured index data, implementing data filtering and desensitization processing based on fine-grained permission control strategy, and finally returning data through a secure transmission channel and using blockchain technology to store audit logs. This method effectively solves the problems of index caliber confusion, serious duplication of development, low query performance, and insufficient data security control, realizes full-link automated processing of index data from definition, production to application, and significantly improves data production efficiency, quality consistency, and security and reliability.

[0089] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0090] If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk.

[0091] The above only describes some embodiments of the present application, and does not limit the protection scope of the present application. Any equivalent device or equivalent flow transformation, or direct or indirect application in other related technical fields, which is made by using the content of the present application specification and drawings, is also included in the patent protection scope of the present application.

Claims

1. A data production and application method based on index management, characterized in that, The method comprises the following steps: receiving an index query request, the index query request comprising a target index identifier, dimension constraint conditions and time range parameters; analyzing and performing semantic verification on the target index identifier based on a global index asset library, to obtain verified index definition information, the global index asset library storing definition information of a plurality of global indexes, each of the global indexes comprising an atomic index, a derived index and a composite index, and the definition information comprising a business scope, technical calculation logic, associated data sources and dimensions of the index; generating a standardized query statement adapted to the structure of a bottom-layer data warehouse based on the index definition information and a index-data mapping relationship, and selecting an optimal data calculation engine to execute the query according to an intelligent routing strategy; applying multi-level cache optimization and parallel computing acceleration to the query execution process to obtain a raw data set, the multi-level cache optimization comprising preheating storage and dynamic updating of high-frequency index query results, and the parallel computing acceleration being distributed processing using an MPP architecture; processing the raw data set in real time according to a preset analysis model to generate structured index data with trend analysis, anomaly detection or attribution inference results; filtering and desensitizing the structured index data based on a fine-grained permission control strategy, and returning only the data content accessible within the permission range of the requester to the requester, and recording the complete data access behavior to an audit log for traceability analysis.

2. The index management based data production and application method according to claim 1, wherein, The global index asset library is generated by the following steps: receiving an index registration request, the index registration request comprising a business scope, technical calculation logic, associated data sources and dimension information of a to-be-registered index; classifying the to-be-registered index into an atomic index, a derived index or a composite index according to the reuse level of the technical calculation logic; if the to-be-registered index is an atomic index, mapping and modeling the to-be-registered index with a measurement field of a core fact table in a data warehouse and a default aggregation method thereof; if the to-be-registered index is a derived index or a composite index, deriving and modeling the to-be-registered index based on an already-defined atomic index or derived index by configuring dimensions, filtering conditions and operation formulas; persisting the mapping relationship between the to-be-registered index output after modeling and data warehouse tables, fields and calculation rules, forming traceable index-data blood relation, and forming index definition information; storing the index definition information into the global index asset library, and assigning a globally unique versioned identifier to the index associated with the index definition information; performing life cycle management on the index definition information in the global index asset library, including automatic inspection and archiving based on access frequency and business relevance.

3. The index management based data production and application method according to claim 1, wherein, Analyzing and performing semantic verification on the target index identifier based on a global index asset library to obtain verified index definition information, comprising: retrieving complete index definition information associated with the identifier from the global index asset library; performing syntax compliance checking on the retrieved index definition information to verify the integrity and standardization of the business scope, technical calculation logic, associated data sources and dimension information of the retrieved index definition information. According to the preset index semantic rule library, the index definition information passing the syntax check is subjected to semantic consistency verification, so as to ensure that there is no contradiction and conflict between the business index and the technical calculation logic; When semantic ambiguity or conflict is detected in the index definition information, an index index abnormality processing procedure is triggered, and a verification report containing specific conflict items and correction suggestions is generated; The index definition information passing all verifications is attached with a digital signature and a time stamp, and verified index definition information trusted by subsequent query execution components is generated.

4. The indicator management based data production and application method of claim 1, wherein, According to the index definition information, a standardized query statement adapted to the underlying data warehouse structure is generated through the index-data mapping relationship, including: The verified index definition information is parsed, and the technical calculation logic, the identification of the associated data source and the dimension information are extracted as query generation elements; Based on the query generation elements, the corresponding data warehouse table, field name and association are determined through the index-data mapping relationship; According to the business index and the technical calculation logic of the index definition information, the dimension constraint condition and the time range parameter are converted into corresponding SQL filter condition clauses; According to the data warehouse hierarchical model optimization principle, the optimal data level is automatically selected and the complete query main body including the select list, the group by grouping and the aggregation function is constructed; An abstract syntax tree-based query optimization algorithm is used to optimize the generated query statement in structure, eliminate redundant calculation and ensure that the syntax conforms to the target calculation engine specification, and output an optimized and completely adapted standardized query statement to the underlying data warehouse structure.

5. The indicator management based data production and application method of claim 1, wherein, According to the intelligent routing strategy, the optimal data calculation engine is selected to execute the query, including: Obtain the query statement characteristics of the standardized query statement, including syntax characteristics, data volume estimation results and complexity score; Based on the query statement characteristics, the preconfigured engine capability profile is matched, which contains the performance and resource consumption characteristics of each data calculation engine for specific query types; Combined with real-time load monitoring data, the query delay, resource utilization and calculation cost are comprehensively evaluated through a multi-objective optimization algorithm to obtain an evaluation result; According to the evaluation result, the optimal target execution engine is selected from the available engine cluster, and the corresponding query task distribution instruction is generated; A session connection between the query task and the execution engine is established to complete the initialization configuration of the query context environment.

6. The indicator management based data production and application method according to claim 5, wherein, Multi-level cache optimization and parallel computing acceleration are applied to the query execution process to obtain the original data set, including: Before query execution, cache matching retrieval is performed based on query statement characteristics and historical execution records. If the cache is hit, the cache data is directly returned; For query requests that do not hit the cache, a cache warming strategy based on a cost model is used to identify high-frequency access patterns and precompute and store the corresponding intermediate results; Complex query tasks are decomposed into multiple parallel executable sub-query units, and an optimal parallel execution plan is generated by a query optimizer; An appropriate amount of computing resources is allocated to each sub-query unit by using a dynamic resource scheduling algorithm, and intelligent cache management is implemented for the intermediate results in the execution process to establish a mapping relationship between query statement characteristics and cache data. The final query result is compressed and stored, and the cache metadata information is updated to generate the original data set.

7. The indicator management based data production and application method according to claim 6, wherein, An appropriate computing resource is allocated to each subquery unit by using a dynamic resource scheduling algorithm, intelligent cache management is implemented for the intermediate results in the execution process, and a mapping relationship between query statement features and cached data is established, including: Based on the computing complexity features and data dependency graph of the subquery units, resource balancing allocation is performed among the computing nodes through a multi-objective optimization algorithm; The resource allocation strategy is dynamically adjusted according to the actual execution progress of each unit; The value of the intermediate result data generated in the execution process is evaluated, and the cache priority is determined based on an access frequency prediction model; A mapping index of query feature vectors and cache data blocks is constructed, and a consistent hashing algorithm is used to realize efficient positioning of the distributed cache; And the cache content is dynamically maintained according to the data access frequency and freshness requirements.

8. The indicator management based data production and application method of claim 1, wherein, The original data set is processed in real time according to the preset analysis model to generate structured index data with trend analysis, anomaly detection or attribution inference results, including: Data quality verification and missing value processing are performed on the original data set to ensure the integrity and accuracy of the input data; According to the business scope of the index definition information, the corresponding analysis model is selected, including a time series prediction model, an isolation forest anomaly detection model or a SHAP attribution analysis model; The preprocessed original data set is input into the selected analysis model to perform batch or streaming calculation, generating preliminary analysis results; The preliminary analysis results are post-processed and optimized, including result smoothing processing, confidence interval calculation and significance annotation; The optimized preliminary analysis results are associated and integrated with the original index data to generate structured index data containing basic data and analysis conclusions.

9. The indicator management based data production and application method of claim 1, wherein, The structured index data is filtered and desensitized based on a fine-grained permission control strategy, including: The identity credentials and access context of the requester are analyzed, and the permission attributes are extracted, including the user identifier, role permission and data access range constraint corresponding to the requester; The access control rule set is matched according to the extracted permission attributes to determine the access range of the target index data for the requester; Based on the determined access range, row and column level data filtering is performed on the structured index data to remove data content outside the requester's permission range; A dynamic desensitization algorithm is applied to the sensitive fields in the retained data to implement differential desensitization processing according to the field sensitivity level and user permission; Audit log records containing data filtering and desensitization processing details are generated, and an association relationship is established with the processed index data to construct a standardized data response package.

10. The indicator management based data production and application method of claim 9, wherein, The data content accessible within the permission range of the requester is returned to the requester, and the complete data access behavior is recorded in the audit log for traceability analysis, including: The standardized data response package is returned to the requester through a secure transmission channel, and data fingerprint verification information is added to the response header of the standardized data response package; Real-time capture of full-link behavior information for this data access, including requester identity, access time, query conditions, returned data volume and processing time consumption; The captured full-link behavior information is associated and integrated with corresponding index definition information and permission control strategies to generate structured audit log records; The audit log is stored by using a blockchain storage technology.

Images