Big data warehouse intelligent analysis method, device, equipment and medium

Abstract

The application discloses a big data warehouse intelligent analysis method and device, equipment and medium, the method comprises the following steps: in response to user analysis demand, and from the initial entry layer of the preset obtains the target resource positioning information corresponding to the user analysis demand; wherein, the initial entry layer is used for storing the resource positioning information of the big data warehouse; the current context is constructed according to the user analysis demand and target resource positioning information;Based on the preset demand coverage inspection strategy, check whether the current context meets the sufficiency condition of analyzing the user analysis demand to obtain the inspection result;If it is determined that the inspection result is a satisfaction result, then the current context is simplified according to the preset context management strategy, and the minimum effective context is obtained;Based on the minimum effective context, analysis reasoning is executed and analysis result is output.The application actively explores data, dynamically constructs the minimum effective context, and significantly improves the accuracy and efficiency of analyzing the big data warehouse.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, equipment and medium for intelligent analysis of big data warehouses. Background Technology

[0002] As enterprise data warehouses continue to expand, their data layering becomes increasingly complex, ETL (Extraction, Transformation, and Loading) processes become lengthy, SQL logic becomes fragmented, and data lineages become deeply coupled. Against this backdrop, the need for AI-assisted data analysis is becoming increasingly urgent. However, traditional AI-assisted data analysis solutions have significant drawbacks, leading to insufficient accuracy and low efficiency when analyzing data warehouses. Specifically, traditional solutions typically inject all information—metadata, complete lineage, ETL scripts, and SQL statements—into the AI ​​model's context at once. This causes the context window to quickly saturate, resulting in information overload and context pollution, leading to decreased reasoning efficiency, increased "answer illusions," and reduced accuracy and response speed. Furthermore, traditional solutions are passive information providers, heavily reliant on pre-retrieved and pushed information, lacking the ability to proactively, hierarchically, and recursively explore and locate the required information based on the analysis objectives, making it difficult to complete complex, multi-step business analysis tasks. Furthermore, in order to adapt to different analysis scenarios (such as lineage analysis, SQL parsing, and ETL interpretation), traditional solutions need to continuously integrate new specialized tools or plugins, which leads to tool bloat and poor scalability, resulting in a bloated system architecture, high maintenance costs, and increased difficulty for AI models in making decisions on tool selection and collaboration. Summary of the Invention

[0003] This invention provides a method, apparatus, device, and medium for intelligent analysis of big data warehouses, aiming to solve the problems of insufficient accuracy and low efficiency in the analysis of data warehouses in the prior art.

[0004] In a first aspect, embodiments of the present invention provide an intelligent analysis method for big data warehouses, comprising:

[0005] In response to user analysis needs, the system obtains target resource location information corresponding to the user analysis needs from a preset initial entry layer; wherein, the initial entry layer is used to store resource location information of the big data warehouse;

[0006] The current context is constructed based on the user analysis requirements and the target resource location information;

[0007] Based on a preset requirement coverage check strategy, check whether the current context meets the sufficiency conditions for analyzing the user's analysis requirements, and obtain the check results.

[0008] If the inspection result is determined to be satisfactory, the current context is simplified according to a preset context management strategy to obtain the minimum effective context.

[0009] The analysis and reasoning are performed based on the minimum effective context, and the analysis results are output.

[0010] Secondly, embodiments of the present invention also provide a big data warehouse intelligent analysis device, the device being used to execute the big data warehouse intelligent analysis method of the first aspect described above, the device comprising:

[0011] A demand response unit is used to respond to user analysis demands and obtain target resource location information corresponding to the user analysis demands from a preset initial entry layer; wherein, the initial entry layer is used to store resource location information of the big data warehouse;

[0012] A context building unit is used to build the current context based on the user analysis requirements and the target resource location information.

[0013] The coverage checking unit is used to check whether the current context meets the sufficiency conditions for analyzing the user analysis requirements based on a preset requirement coverage checking strategy, so as to obtain the checking results;

[0014] The context simplification element is used to simplify the current context according to a preset context management strategy to obtain the minimum effective context if the check result is determined to be a satisfactory result.

[0015] The analysis and reasoning unit is used to perform analysis and reasoning based on the minimum effective context and output the analysis results.

[0016] Thirdly, embodiments of the present invention provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the intelligent analysis method for big data warehouses described in the first aspect.

[0017] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the big data warehouse intelligent analysis method described in the first aspect.

[0018] This invention provides a method, apparatus, device, and medium for intelligent analysis of big data warehouses. The method includes responding to user analysis requests and acquiring target resource location information corresponding to the user's analysis requests from a preset initial entry layer; wherein the initial entry layer stores resource location information of the big data warehouse; constructing a current context based on the user analysis requests and target resource location information; checking whether the current context meets the sufficiency conditions for analyzing the user's analysis requests based on a preset requirement coverage check strategy, to obtain a check result; if the check result is determined to be satisfactory, simplifying the current context according to a preset context management strategy to obtain a minimum effective context; performing analysis and reasoning based on the minimum effective context and outputting the analysis results. This invention significantly improves the accuracy and efficiency of big data warehouse analysis by actively exploring data and dynamically constructing a minimum effective context. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a big data warehouse intelligent analysis method provided in an embodiment of the present invention;

[0021] Figure 2 This is a schematic block diagram of a big data warehouse intelligent analysis device provided in an embodiment of the present invention;

[0022] Figure 3 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0026] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0027] Please see Figure 1 , Figure 1 This is a flowchart illustrating a big data warehouse intelligent analysis method according to an embodiment of the present invention. The big data warehouse intelligent analysis method provided in this embodiment of the present invention, as follows... Figure 1 As shown, the intelligent analysis method for big data warehouses includes the following steps S11 to S15.

[0028] S11. Responding to user analysis needs, the system obtains target resource location information corresponding to the user analysis needs from a preset initial entry layer; wherein, the initial entry layer is used to store resource location information of the big data warehouse.

[0029] In this embodiment, by setting an initial entry layer to store the resource location information of the big data warehouse, when a user's analysis request is received, the system actively explores and extracts the target resource location information related to the user's analysis request from the initial entry layer. Thus, even without a complete understanding of the big data warehouse, the system can still provide all the location information needed for subsequent exploration through the target resource location information. This eliminates the need for the user to inject all information at once, and allows for proactive and recursive exploration and location of the required information based on the user's analysis needs.

[0030] In one embodiment, obtaining target resource location information corresponding to the user's analysis needs from a preset initial entry layer includes:

[0031] Semantic parsing is performed on the user analysis requirements to obtain the target data table name;

[0032] Obtain target resource location information related to the target data table name from the initial entry layer; wherein, the target resource location information includes data warehouse hierarchical topology information, core table metadata index information, data lineage root node information, ETL task overview information, and business SQL template index information related to the target data table name.

[0033] In this embodiment, the initial entry layer serves as the system information entry point, specifically including the data warehouse hierarchical topology entry point, the core table metadata index entry point, the data lineage root node entry point, the ETL task overview entry point, and the business SQL template index entry point. The data warehouse hierarchical topology entry point records the hierarchical structure of the big data warehouse and the upstream and downstream dependencies between each level. These dependencies characterize the flow of data between levels. For example, the hierarchical structure may include ODS (Operational Data Store), DWD (Data Warehouse Detail), DWS (Data Warehouse Summary), and ADS (Application Data Service), with the dependency relationship being ODS→DWD→DWS→ADS. The core table metadata index entry point records the table name, its level, a summary of the number of fields, a brief description of the table's purpose, and the corresponding metadata access path for each core data table in the big data warehouse. It does not include the complete definition and constraint information of each field. The data lineage root node entry records the node identifiers of each core data table in the lineage graph of the big data warehouse and a list of its direct upstream table names, but does not include the complete multi-level lineage chain. The ETL task overview entry records the ETL task names, task status summaries, and corresponding ETL script storage paths (i.e., the storage paths of ETL script files) related to each core data table, but does not include the actual code content of the ETL scripts. The business SQL template index entry records the core business SQL template names associated with each core data table, a brief description of the template's purpose, and the corresponding SQL template storage path (i.e., the storage path of SQL script files), but does not include the complete content of the SQL statements. Therefore, the resource location information of the big data warehouse is the information recorded in each entry layer of the initial entry layer, which only includes resource location addresses (such as metadata access paths, ETL script storage paths, etc.) and summary descriptions (such as table names, field quantity summaries, etc.), and does not include index-style information of full-scale detailed data.

[0034] In practical applications, after receiving a user's analysis request, the request is first semantically parsed to extract the target data table names. Then, based on the data warehouse hierarchical topology entry, core table metadata index entry, data lineage root node entry, ETL task overview entry, and business SQL template index entry in the initial entry layer, the data warehouse hierarchical topology information, core table metadata index information, data lineage root node information, ETL task overview information, and business SQL template index information related to the target data table name are matched and read from each entry, and the target resource location information is formed. For example, if a user's analysis requirement is "to analyze the data source, ETL processing flow, and SQL calculation logic of the ADS layer sales summary table," semantic parsing is performed on this user's analysis requirement. The extracted target data table name is "ADS layer sales summary table." Based on this target data table name, the following target resource location information is matched and read from the initial entry layer: Data warehouse layered topology information: the hierarchical structure includes ODS, DWD, DWS, and ADS, and the dependency relationship between the layers is ODS→DWD→DWS→ADS; Core table metadata index information: the table name, layer, field quantity summary, table purpose summary, and corresponding metadata access path of the ADS layer sales summary table; Data lineage root node information: the node identifier of the ADS layer sales summary table in the lineage relationship graph and its direct upstream table name list; ETL task overview information: the ETL task name, task status summary, and corresponding ETL script storage path related to the ADS layer sales summary table; Business SQL template index information: the core business SQL template name, template purpose summary, and corresponding SQL template storage path associated with the ADS layer sales summary table.

[0035] S12. Construct the current context based on the user analysis requirements and the target resource location information.

[0036] S13. Based on a preset requirement coverage check strategy, check whether the current context meets the sufficiency conditions for analyzing the user analysis requirements, so as to obtain the check result.

[0037] In this embodiment, user analysis requirements and target resource location information are combined into the current context. Then, based on the requirement coverage check strategy, it is checked whether the current context meets the sufficiency conditions for analyzing user analysis requirements. That is, it is checked whether the current context contains the key information required to complete the analysis of user analysis requirements, thereby obtaining the check result.

[0038] More specifically, in one embodiment, step S13 includes: decomposing the user analysis requirements into several analysis sub-tasks; checking whether the current context contains the key information required to complete each analysis sub-task, so as to obtain the check results.

[0039] In this embodiment, firstly, semantic parsing is performed on the user analysis requirements to extract several analysis dimension keywords, and the user analysis requirements are decomposed into analysis sub-tasks corresponding to each analysis dimension keyword. Then, the current context is checked one by one to see if it contains the key information required to complete each analysis sub-task. If the current context contains the key information required to complete each analysis sub-task, that is, the current context meets the sufficiency condition for analyzing the user analysis requirements, the current check result is determined to be a satisfactory result. If the current context lacks the key information required to complete any analysis sub-task, that is, the current context does not meet the sufficiency condition for analyzing the user analysis requirements, the current check result is determined to be a unsatisfactory result, and the identified missing information set is obtained, so that the required information can be actively explored based on the missing information set, thereby achieving the purpose of autonomously locating, decomposing layer by layer, and recursively obtaining information based on analysis requirements.

[0040] For example, if a user's analysis requirement is "to analyze the data source, ETL processing flow, and SQL calculation logic of the ADS layer sales summary table", three analysis dimension keywords can be extracted: "data source", "ETL processing", and "SQL logic". Based on these three analysis dimension keywords, the user's analysis requirement can be decomposed into three corresponding analysis sub-tasks: analysis sub-task A: data source tracing, analysis sub-task B: ETL processing flow analysis, and analysis sub-task C: SQL calculation logic interpretation. The current context is checked one by one to see if it contains the key information required to complete analysis subtasks A, B, and C. For analysis subtask A, the current context only contains the node identifier (i.e., the root node identifier) ​​of the ADS layer sales summary table in the lineage graph and its direct upstream table name list, lacking the field metadata of the ADS layer sales summary table and its upstream tables, and the complete multi-level lineage chain. For analysis subtask B, the current context only contains the ETL task name, task status summary, and corresponding ETL script storage path, lacking the specific ETL processing logic. For analysis subtask C, the current context only contains the core business SQL template name, a brief description of the template's purpose, and the corresponding SQL template storage path, lacking the specific SQL calculation rules. For these three analysis subtasks, the current context contains missing key information required to complete each analysis subtask, thus the current check result is determined to be unsatisfactory, and the "missing field metadata of the ADS layer sales summary table and its upstream tables, the complete multi-level lineage chain, the missing ETL processing logic, and the missing SQL calculation rules" are combined into a missing information set.

[0041] In one embodiment, after step S13, the method further includes:

[0042] If the inspection result is determined to be a non-satisfactory result, the corresponding data warehouse details are obtained according to the missing information set corresponding to the non-satisfactory result and the preset information exploration strategy.

[0043] The current context and the data warehouse details are combined to form a correction context, and the correction context is simplified according to the context management strategy to obtain an optimized context;

[0044] The steps involve updating the current context using the optimized context and returning to execute the preset requirement coverage check strategy to check whether the current context meets the sufficiency conditions for analyzing the user analysis requirements, so as to obtain the check results.

[0045] In this embodiment, if the inspection result is determined to be unsatisfactory, it indicates that the current context does not meet the sufficiency conditions for analyzing the user's analysis needs. Therefore, proactive exploration to supplement details is initiated. Specifically, based on the identified missing information set and a preset information exploration strategy, the required information is actively explored to obtain the corresponding data warehouse details. The current context and the data warehouse details are combined to form a corrected context. This corrected context is then streamlined according to a context management strategy to eliminate irrelevant, redundant, and overlapping information, resulting in an optimized context. The corrected context is used to update the current context, ensuring that the context after each round of exploration and supplementation is always the minimum effective set for completing the current analysis task. Then, the process returns to step S13 until the current context contains the key information required to complete the analysis of the user's analysis needs. This achieves autonomous, layered exploration of the big data warehouse, enabling autonomous location, layer-by-layer decomposition, and recursive information acquisition based on analysis needs.

[0046] In one embodiment, obtaining the corresponding data warehouse details based on the missing information set corresponding to the unsatisfactory result and a preset information exploration strategy includes:

[0047] If it is determined that the missing information set contains missing information of the field metadata type, then the metadata access path corresponding to the missing information of the field metadata type is obtained from the initial entry layer, and the corresponding complete field definition is obtained based on the metadata access path and the preset metadata on-demand reading tool, and the complete field definition is used as the corresponding sub-detail information;

[0048] If it is determined that the missing information set contains missing information of blood relationship type, then a hierarchical blood relationship subgraph is obtained based on the blood relationship root node identifier in the current context, the preset traversal level depth and the preset data blood relationship hierarchical traversal tool, and the hierarchical blood relationship subgraph is used as the corresponding sub-detail information;

[0049] If it is determined that the missing information set contains missing information of the computational logic type, then the SQL computational logic parsing result is obtained based on the SQL template storage path in the current context and the preset SQL syntax logic parsing tool, and the SQL computational logic parsing result is used as the corresponding sub-detail information;

[0050] If it is determined that the missing information set contains missing information of the processing flow type, then the ETL core logic fragment list is obtained based on the ETL script storage path in the current context and the preset ETL task fragment extraction tool, and the ETL core logic fragment list is used as the corresponding sub-detail information.

[0051] All the obtained sub-detail information is combined to form the data warehouse detail information.

[0052] In this embodiment, the information exploration strategy is used to extract the required sub-detail information by invoking corresponding tools in a preset active exploration tool layer based on the type of missing information in the missing information set. All the obtained sub-detail information is then combined into data warehouse detail information to supplement the current context. The missing information type is one of the following: field metadata type, lineage relationship type, computational logic type, and processing flow type. The active exploration tool layer includes a metadata on-demand reading tool, a data lineage hierarchical traversal tool, an SQL syntax logic parsing tool, and an ETL task fragment extraction tool, corresponding to the field metadata type, lineage relationship type, computational logic type, and processing flow type, respectively. By invoking the common metadata on-demand reading tool, data lineage hierarchical traversal tool, SQL syntax logic parsing tool, and ETL task fragment extraction tool in the active exploration tool layer based on the type of missing information, the required sub-detail information is extracted without continuously adding a large number of dedicated tools to adapt to different analysis scenarios, thus solving the problems of tool bloat and poor scalability in existing solutions.

[0053] Specifically, if the missing information set is determined to contain missing information of a field metadata type, the metadata on-demand reading tool is invoked. The metadata access path corresponding to the missing information of the field metadata type is obtained from the initial entry layer and input into the metadata on-demand reading tool. This allows the tool to obtain the complete field definition as the corresponding sub-detail information. Specifically, the metadata on-demand reading tool locates the metadata storage location based on the metadata access path, extracts detailed definition information such as the data type, business meaning, value constraints, and null value ratio of the specified required fields, and assembles and outputs the complete field definition. For example, if the missing information set contains missing information of the field metadata type "missing field metadata of the ADS layer sales summary table and its upstream table," the metadata access paths corresponding to the ADS layer sales summary table and its upstream table are obtained from the initial entry layer and input into the metadata on-demand reading tool. The tool then obtains the detailed definition information of each field (such as amount, quantity, region, and date) corresponding to the ADS layer sales summary table and its upstream table, and assembles the complete field definition.

[0054] If it is determined that the missing information set contains missing information of bloodline relationship type, the data bloodline hierarchical traversal tool is invoked. The bloodline root node identifier in the current context and the preset traversal level depth are input into the data bloodline hierarchical traversal tool to obtain a hierarchical bloodline subgraph as the corresponding sub-detail information. Specifically, the data bloodline hierarchical traversal tool starts from the bloodline root node identifier and traces upstream or downstream dependency tables layer by layer according to the preset traversal level depth (such as traversing upwards by N levels of bloodline depth). It records the table name, the level to which it belongs, and the upstream and downstream dependencies between each level within the preset traversal level depth, and generates a hierarchical bloodline subgraph containing multi-level bloodline links accordingly. For example, if the missing information set contains the missing information of the lineage relationship type as "the complete multi-level lineage link of the ADS layer sales summary table", and the root node identifier in the current context is "the node identifier of the ADS layer sales summary table in the lineage relationship graph", and the preset traversal level depth is "traversing upwards by 2 levels of lineage depth", then input "the node identifier of the ADS layer sales summary table in the lineage relationship graph" and "traversing upwards by 2 levels of lineage depth" into the data lineage hierarchical traversal tool, and obtain a hierarchical lineage subgraph containing "DWD layer sales fact table → DWS layer sales detail table → ADS layer sales summary table" through the data lineage hierarchical traversal tool.

[0055] If the missing information set is determined to contain missing information of a computational logic type (such as "missing specific SQL calculation rules"), the SQL syntax logic parsing tool is invoked. The SQL template storage path in the current context is input into the SQL syntax logic parsing tool to obtain the SQL computational logic parsing result as the corresponding sub-detail information. Specifically, the SQL syntax logic parsing tool performs lexical analysis and syntax tree construction on the SQL statements in the SQL template storage path, extracting structured information such as the SELECT field list, FROM data source table, JOIN join conditions, WHERE filter conditions, GROUP BY grouping fields, and aggregate functions. Based on this structured information, it generates the SQL computational logic parsing result, which includes field mapping relationships, table join relationships, and a summary of computational logic.

[0056] If the missing information set is determined to contain missing information of the processing flow type (such as missing specific ETL processing logic), the ETL task fragment extraction tool is invoked. Based on the current context, several keywords of interest (such as "cleaning," "aggregation," "deduplication," etc.) are generated. The ETL script storage path and several keywords of interest in the current context are input into the ETL task fragment extraction tool to obtain a list of ETL core logic fragments as corresponding sub-detail information. Specifically, the ETL task fragment extraction tool reads the ETL script file in the ETL script storage path, segments the ETL script file according to the task execution stage, filters comment lines and log output statements, extracts the core processing logic fragment corresponding to each keyword of interest, and generates a list of ETL core logic fragments based on the extracted core processing logic fragments, the task execution stage to which each core processing logic fragment belongs, and the processing purpose.

[0057] In one embodiment, after obtaining the hierarchical pedigree subgraph based on the pedigree root node identifier in the current context, a preset traversal level depth, and a preset data pedigree hierarchical traversal tool, and using the hierarchical pedigree subgraph as the corresponding sub-detail information, the method further includes:

[0058] If the preset path analysis triggering conditions are determined based on the current context, then the preset data lineage analysis sub-agent is used to perform path analysis on the hierarchical lineage subgraph to obtain a lineage critical path analysis report.

[0059] Add the aforementioned critical path analysis report to the corresponding sub-detail information and update it.

[0060] In this embodiment, the information exploration strategy is further used to, after obtaining the corresponding sub-detail information by calling the tools in the active exploration tool layer, and determining, based on the current context, that the corresponding triggering condition is met, invoke the corresponding analysis sub-agent in the preset analysis sub-agent layer to execute the corresponding analysis task, thereby obtaining the corresponding analysis report to update the sub-detail information, so that the analysis report can be subsequently added to the current context. Specifically, after each analysis sub-agent completes its analysis, it only adds the conclusive analysis report to the corresponding sub-detail information, ensuring that intermediate reasoning processes are not injected into the current context, thus avoiding context pollution. By exploring the required sub-detail information through the tools in the active exploration tool layer and the analysis sub-agents in the analysis sub-agent layer, active information exploration according to a progressive disclosure approach is achieved.

[0061] Specifically, if the current context determines that the preset path analysis trigger conditions are met, indicating that lineage path analysis and risk assessment are required, then the data lineage analysis sub-agent in the analysis sub-agent layer further performs path analysis on the hierarchical lineage subgraph to obtain a lineage critical path analysis report. This report is then added to and updated in the corresponding sub-detail information to facilitate the subsequent addition of the hierarchical lineage subgraph and the lineage critical path analysis report to the current context. More specifically, lineage analysis concerns (such as locating the original data source of a certain indicator) are generated based on the current context, and the hierarchical lineage subgraph and the concerns are input into the data lineage analysis sub-agent in the analysis sub-agent layer for analysis. The data lineage analysis sub-agent, based on the concerns, identifies key dependency paths, branching nodes, and convergence nodes in the hierarchical lineage subgraph, assesses the data transmission integrity on each path, and generates a lineage critical path analysis report based on the identification and assessment results. The Critical Path Analysis Report includes the core dependency chain from the target data table to its source data table, descriptions of the data processing roles of each node, and potential risk points of lineage breakage.

[0062] In one embodiment, after obtaining the SQL calculation logic parsing result based on the SQL template storage path in the current context and a preset SQL syntax logic parsing tool, and using the SQL calculation logic parsing result as the corresponding sub-detail information, the method further includes:

[0063] If the preset logical translation triggering condition is determined based on the current context, the preset SQL logic analysis sub-agent is used to perform business semantic analysis on the SQL calculation logic parsing result to obtain an SQL business semantic analysis report.

[0064] Add the SQL business semantic analysis report to the corresponding sub-detail information and update it.

[0065] In this embodiment, if the preset logic translation trigger condition is met based on the current context, indicating that the SQL technical logic needs to be translated into business semantics, then the SQL logic analysis sub-agent in the analysis sub-agent layer further performs business semantic analysis on the SQL calculation logic parsing result to obtain an SQL business semantic analysis report. Then, the SQL business semantic analysis report is added to the corresponding sub-detail information and updated so that both the SQL calculation logic parsing result and the SQL business semantic analysis report can be subsequently added to the current context.

[0066] More specifically, based on the current context, SQL logic analysis concerns (such as understanding the calculation method of a certain indicator) are generated. The SQL calculation logic parsing results and SQL logic analysis concerns are input into the SQL logic analysis sub-agent in the analysis sub-agent layer for analysis and processing. Specifically, the SQL logic analysis sub-agent, based on the SQL logic analysis concerns, performs business semantic inference on the field mapping relationships, table relationships, and calculation logic summaries in the SQL calculation logic parsing results, translating the technical-level SQL logic into a business-level calculation rule description. Based on the translated calculation rule description, an SQL business semantic analysis report is generated. This report includes the business calculation method for each output field, explanations of business rules for data filtering and aggregation, and potential logical risks (such as implicit type conversions and omissions in null value handling).

[0067] In one embodiment, after obtaining the ETL core logic fragment list based on the ETL script storage path in the current context and a preset ETL task fragment extraction tool, and using the ETL core logic fragment list as the corresponding sub-detail information, the method further includes:

[0068] If the preset process analysis triggering conditions are determined to be met based on the current context, the preset ETL process analysis sub-agent is used to analyze the list of ETL core logic segments to obtain an ETL process analysis report.

[0069] Add the ETL process analysis report to the corresponding sub-detail information and update it.

[0070] In this embodiment, if the preset process analysis triggering conditions are met based on the current context, indicating that an integrity assessment and risk analysis of the ETL process is required, the ETL process analysis sub-agent in the sub-agent layer is further used to analyze the list of ETL core logic segments to obtain an ETL process analysis report. Then, the ETL process analysis report is added to the corresponding sub-detail information and updated, so that the list of ETL core logic segments and the ETL process analysis report can be subsequently added to the current context.

[0071] More specifically, based on the current context, ETL process analysis concerns (such as summarizing data cleaning rules) are generated. The list of core ETL logic segments and these concerns are then input into the ETL process analysis sub-agent in the analysis sub-agent layer for analysis. The ETL process analysis sub-agent, based on these concerns, reconstructs the execution order of the core ETL logic segments in the list, analyzes the data input sources, transformation rules, and output destinations at each processing stage, assesses the coverage of data quality control measures, and generates an ETL process analysis report. This report includes a description of the processing steps for each stage, a summary of data transformation rules, and potential data quality risks.

[0072] S14. If the inspection result is determined to be satisfactory, the current context is simplified according to the preset context management strategy to obtain the minimum effective context.

[0073] In this embodiment, if the check result is determined to be satisfactory, it indicates that the current context meets the sufficiency conditions for analyzing the user's analysis needs. Then, according to a preset context management strategy, the current context is simplified to dynamically construct the minimum effective context. This ensures that the context is always the minimum effective set for completing the current analysis task, avoiding context overload and pollution, and improving the accuracy and response speed of analysis and reasoning. More specifically, the context management strategy uses the user's analysis needs as a benchmark to perform information filtering and granularity normalization on the current context. This eliminates irrelevant, redundant, and overlapping information, thereby removing information entries not directly related to the user's analysis needs (such as side nodes not on the core dependency path in lineage traversal), removing duplicate information, and replacing covered coarse-grained old information with finer-grained new information. After updating, the minimum effective context is obtained.

[0074] S15. Perform analysis and reasoning based on the minimum effective context and output the analysis results.

[0075] In this embodiment, analysis and reasoning are performed based on the minimum effective context to complete the analysis and reasoning of user analysis requirements, thereby obtaining analysis results. This achieves intelligent analysis of the big data warehouse, and the analysis results can assist data development and other related personnel in data development and analysis. Specifically, step S15 includes: classifying and integrating various types of information in the minimum effective context according to several analysis sub-tasks in the user analysis requirements to obtain the effective context information corresponding to each analysis sub-task; for each analysis sub-task, reasoning is performed based on the corresponding effective context information to obtain the analysis conclusion corresponding to each analysis sub-task; and the analysis conclusions corresponding to each analysis sub-task are summarized to form a complete analysis result. For example, for the data source category sub-task, the relevant lineage data in the minimum effective context is classified and integrated into the corresponding effective context information, and the data flow path is sorted out along the multi-level lineage links in the effective context information to generate source link descriptions, so as to obtain the analysis conclusion corresponding to the data source category sub-task. For ETL process subtasks, the relevant ETL processes and SQL rules in the minimum effective context are categorized and integrated into corresponding effective context information. Based on this effective context information, ETL processing steps and calculation methods are explained to obtain the analytical conclusions corresponding to this ETL process subtask. For risk identification subtasks, the relevant metadata constraints, ETL quality control measures, and SQL logic risk points in the minimum effective context are categorized and integrated into corresponding effective context information. Based on this effective context information, potential risk warnings are generated to obtain the analytical conclusions corresponding to this risk identification subtask.

[0076] The intelligent analysis method for big data warehouses provided in this invention includes: responding to user analysis needs and obtaining target resource location information corresponding to the user analysis needs from a preset initial entry layer; wherein, the initial entry layer is used to store resource location information of the big data warehouse; constructing a current context based on the user analysis needs and the target resource location information; checking whether the current context meets the sufficiency conditions for analyzing the user analysis needs based on a preset demand coverage check strategy, to obtain a check result; if the check result is determined to be satisfactory, simplifying the current context according to a preset context management strategy to obtain a minimum effective context; performing analysis and reasoning based on the minimum effective context and outputting the analysis results. This invention significantly improves the accuracy and efficiency of big data warehouse analysis by actively exploring data and dynamically constructing a minimum effective context, without relying on the user to inject all information at once.

[0077] This invention also provides a big data warehouse intelligent analysis device, which is used to execute any of the aforementioned big data warehouse intelligent analysis methods. Specifically, please refer to... Figure 2 , Figure 2This is a schematic block diagram of a big data warehouse intelligent analysis device provided in an embodiment of the present invention. The big data warehouse intelligent analysis device provided in this embodiment of the present invention includes: a demand response unit 11, a context construction unit 12, a coverage check unit 13, a context refinement unit 14, and an analysis and reasoning unit 15.

[0078] The demand response unit 11 is used to respond to user analysis needs and obtain target resource location information corresponding to the user analysis needs from a preset initial entry layer; wherein, the initial entry layer is used to store resource location information of the big data warehouse.

[0079] In this embodiment, by setting an initial entry layer to store the resource location information of the big data warehouse, when a user's analysis request is received, the system actively explores and extracts the target resource location information related to the user's analysis request from the initial entry layer. Thus, even without a complete understanding of the big data warehouse, the system can still provide all the location information needed for subsequent exploration through the target resource location information. This eliminates the need for the user to inject all information at once, and allows for proactive and recursive exploration and location of the required information based on the user's analysis needs.

[0080] In one embodiment, when the demand response unit 11 performs the step of obtaining the target resource location information corresponding to the user analysis demand from the preset initial entry layer, it is specifically used for:

[0081] Semantic parsing is performed on the user analysis requirements to obtain the target data table name;

[0082] Obtain target resource location information related to the target data table name from the initial entry layer; wherein, the target resource location information includes data warehouse hierarchical topology information, core table metadata index information, data lineage root node information, ETL task overview information, and business SQL template index information related to the target data table name.

[0083] In this embodiment, the initial entry layer serves as the system information entry point, specifically including the data warehouse hierarchical topology entry point, the core table metadata index entry point, the data lineage root node entry point, the ETL task overview entry point, and the business SQL template index entry point. The data warehouse hierarchical topology entry point records the hierarchical structure of the big data warehouse and the upstream and downstream dependencies between each level. These dependencies characterize the flow of data between levels. For example, the hierarchical structure may include ODS (Operational Data Store), DWD (Data Warehouse Detail), DWS (Data Warehouse Summary), and ADS (Application Data Service), with the dependency relationship being ODS→DWD→DWS→ADS. The core table metadata index entry point records the table name, its level, a summary of the number of fields, a brief description of the table's purpose, and the corresponding metadata access path for each core data table in the big data warehouse. It does not include the complete definition and constraint information of each field. The data lineage root node entry records the node identifiers of each core data table in the lineage graph of the big data warehouse and a list of its direct upstream table names, but does not include the complete multi-level lineage chain. The ETL task overview entry records the ETL task names, task status summaries, and corresponding ETL script storage paths (i.e., the storage paths of ETL script files) related to each core data table, but does not include the actual code content of the ETL scripts. The business SQL template index entry records the core business SQL template names associated with each core data table, a brief description of the template's purpose, and the corresponding SQL template storage path (i.e., the storage path of SQL script files), but does not include the complete content of the SQL statements. Therefore, the resource location information of the big data warehouse is the information recorded in each entry layer of the initial entry layer, which only includes resource location addresses (such as metadata access paths, ETL script storage paths, etc.) and summary descriptions (such as table names, field quantity summaries, etc.), and does not include index-style information of full-scale detailed data.

[0084] In practical applications, after receiving a user's analysis request, the request is first semantically parsed to extract the target data table names. Then, based on the data warehouse hierarchical topology entry, core table metadata index entry, data lineage root node entry, ETL task overview entry, and business SQL template index entry in the initial entry layer, the data warehouse hierarchical topology information, core table metadata index information, data lineage root node information, ETL task overview information, and business SQL template index information related to the target data table name are matched and read from each entry, and the target resource location information is formed. For example, if a user's analysis requirement is "to analyze the data source, ETL processing flow, and SQL calculation logic of the ADS layer sales summary table," semantic parsing is performed on this user's analysis requirement. The extracted target data table name is "ADS layer sales summary table." Based on this target data table name, the following target resource location information is matched and read from the initial entry layer: Data warehouse layered topology information: the hierarchical structure includes ODS, DWD, DWS, and ADS, and the dependency relationship between the layers is ODS→DWD→DWS→ADS; Core table metadata index information: the table name, layer, field quantity summary, table purpose summary, and corresponding metadata access path of the ADS layer sales summary table; Data lineage root node information: the node identifier of the ADS layer sales summary table in the lineage relationship graph and its direct upstream table name list; ETL task overview information: the ETL task name, task status summary, and corresponding ETL script storage path related to the ADS layer sales summary table; Business SQL template index information: the core business SQL template name, template purpose summary, and corresponding SQL template storage path associated with the ADS layer sales summary table.

[0085] The context building unit 12 is used to build the current context based on the user analysis requirements and the target resource location information.

[0086] The coverage checking unit 13 is used to check whether the current context meets the sufficiency conditions for analyzing the user analysis requirements based on a preset requirement coverage checking strategy, so as to obtain the checking result.

[0087] In this embodiment, the context construction unit 12 combines user analysis requirements and target resource location information into the current context. Then, the coverage check unit 13 checks whether the current context meets the sufficiency conditions for analyzing user analysis requirements based on the requirement coverage check strategy. That is, it checks whether the current context contains the key information required to complete the analysis of user analysis requirements, thereby obtaining the check result.

[0088] More specifically, in one embodiment, the coverage checking unit 13 is specifically used to: decompose the user analysis requirements into several analysis sub-tasks; check one by one whether the current context contains the key information required to complete each analysis sub-task, so as to obtain the check result.

[0089] In this embodiment, firstly, semantic parsing is performed on the user analysis requirements to extract several analysis dimension keywords, and the user analysis requirements are decomposed into analysis sub-tasks corresponding to each analysis dimension keyword. Then, the current context is checked one by one to see if it contains the key information required to complete each analysis sub-task. If the current context contains the key information required to complete each analysis sub-task, that is, the current context meets the sufficiency condition for analyzing the user analysis requirements, the current check result is determined to be a satisfactory result. If the current context lacks the key information required to complete any analysis sub-task, that is, the current context does not meet the sufficiency condition for analyzing the user analysis requirements, the current check result is determined to be a unsatisfactory result, and the identified missing information set is obtained, so that the required information can be actively explored based on the missing information set, thereby achieving the purpose of autonomously locating, decomposing layer by layer, and recursively obtaining information based on analysis requirements.

[0090] For example, if a user's analysis requirement is "to analyze the data source, ETL processing flow, and SQL calculation logic of the ADS layer sales summary table", three analysis dimension keywords can be extracted: "data source", "ETL processing", and "SQL logic". Based on these three analysis dimension keywords, the user's analysis requirement can be decomposed into three corresponding analysis sub-tasks: analysis sub-task A: data source tracing, analysis sub-task B: ETL processing flow analysis, and analysis sub-task C: SQL calculation logic interpretation. The current context is checked one by one to see if it contains the key information required to complete analysis subtasks A, B, and C. For analysis subtask A, the current context only contains the node identifier (i.e., the root node identifier) ​​of the ADS layer sales summary table in the lineage graph and its direct upstream table name list, lacking the field metadata of the ADS layer sales summary table and its upstream tables, and the complete multi-level lineage chain. For analysis subtask B, the current context only contains the ETL task name, task status summary, and corresponding ETL script storage path, lacking the specific ETL processing logic. For analysis subtask C, the current context only contains the core business SQL template name, a brief description of the template's purpose, and the corresponding SQL template storage path, lacking the specific SQL calculation rules. For these three analysis subtasks, the current context contains missing key information required to complete each analysis subtask, thus the current check result is determined to be unsatisfactory, and the "missing field metadata of the ADS layer sales summary table and its upstream tables, the complete multi-level lineage chain, the missing ETL processing logic, and the missing SQL calculation rules" are combined into a missing information set.

[0091] In one embodiment, after executing the step of checking whether the current context meets the sufficiency conditions for analyzing the user analysis requirements based on the preset requirement coverage check strategy to obtain the check result, the coverage check unit 13 is further configured to:

[0092] If the inspection result is determined to be a non-satisfactory result, the corresponding data warehouse details are obtained according to the missing information set corresponding to the non-satisfactory result and the preset information exploration strategy.

[0093] The current context and the data warehouse details are combined to form a correction context, and the correction context is simplified according to the context management strategy to obtain an optimized context;

[0094] The steps involve updating the current context using the optimized context and returning to execute the preset requirement coverage check strategy to check whether the current context meets the sufficiency conditions for analyzing the user analysis requirements, so as to obtain the check results.

[0095] In this embodiment, if the check result is determined to be unsatisfactory, it indicates that the current context does not meet the sufficiency conditions for analyzing the user's analysis needs. Therefore, proactive exploration to supplement details is initiated. Specifically, based on the identified missing information set and a preset information exploration strategy, the required information is actively explored to obtain the corresponding data warehouse details. The current context and the data warehouse details are combined to form a corrected context. This corrected context is then streamlined according to a context management strategy to eliminate irrelevant, redundant, and overlapping information, resulting in an optimized context. The corrected context is used to update the current context, ensuring that the context after each round of exploration and supplementation of details is always the minimum effective set for completing the current analysis task. Then, the process returns to execute the preset requirement coverage check strategy to check whether the current context meets the sufficiency conditions for analyzing the user's analysis needs, obtaining the check result. This process continues until the current context contains the key information required to complete the analysis of the user's analysis needs, achieving autonomous hierarchical exploration of the big data warehouse and realizing the purpose of autonomous positioning, layer-by-layer decomposition, and recursive information acquisition based on analysis needs.

[0096] In one embodiment, when the coverage checking unit 13 performs the step of obtaining the corresponding data warehouse details based on the missing information set corresponding to the unsatisfactory result and a preset information exploration strategy, it is specifically used for:

[0097] If it is determined that the missing information set contains missing information of the field metadata type, then the metadata access path corresponding to the missing information of the field metadata type is obtained from the initial entry layer, and the corresponding complete field definition is obtained based on the metadata access path and the preset metadata on-demand reading tool, and the complete field definition is used as the corresponding sub-detail information;

[0098] If it is determined that the missing information set contains missing information of blood relationship type, then a hierarchical blood relationship subgraph is obtained based on the blood relationship root node identifier in the current context, the preset traversal level depth and the preset data blood relationship hierarchical traversal tool, and the hierarchical blood relationship subgraph is used as the corresponding sub-detail information;

[0099] If it is determined that the missing information set contains missing information of the computational logic type, then the SQL computational logic parsing result is obtained based on the SQL template storage path in the current context and the preset SQL syntax logic parsing tool, and the SQL computational logic parsing result is used as the corresponding sub-detail information;

[0100] If it is determined that the missing information set contains missing information of the processing flow type, then the ETL core logic fragment list is obtained based on the ETL script storage path in the current context and the preset ETL task fragment extraction tool, and the ETL core logic fragment list is used as the corresponding sub-detail information.

[0101] All the obtained sub-detail information is combined to form the data warehouse detail information.

[0102] In this embodiment, the information exploration strategy is used to extract the required sub-detail information by invoking corresponding tools in a preset active exploration tool layer based on the type of missing information in the missing information set. All the obtained sub-detail information is then combined into data warehouse detail information to supplement the current context. The missing information type is one of the following: field metadata type, lineage relationship type, computational logic type, and processing flow type. The active exploration tool layer includes a metadata on-demand reading tool, a data lineage hierarchical traversal tool, an SQL syntax logic parsing tool, and an ETL task fragment extraction tool, corresponding to the field metadata type, lineage relationship type, computational logic type, and processing flow type, respectively. By invoking the common metadata on-demand reading tool, data lineage hierarchical traversal tool, SQL syntax logic parsing tool, and ETL task fragment extraction tool in the active exploration tool layer based on the type of missing information, the required sub-detail information is extracted without continuously adding a large number of dedicated tools to adapt to different analysis scenarios, thus solving the problems of tool bloat and poor scalability in existing solutions.

[0103] Specifically, if the missing information set is determined to contain missing information of a field metadata type, the metadata on-demand reading tool is invoked. The metadata access path corresponding to the missing information of the field metadata type is obtained from the initial entry layer and input into the metadata on-demand reading tool. This allows the tool to obtain the complete field definition as the corresponding sub-detail information. Specifically, the metadata on-demand reading tool locates the metadata storage location based on the metadata access path, extracts detailed definition information such as the data type, business meaning, value constraints, and null value ratio of the specified required fields, and assembles and outputs the complete field definition. For example, if the missing information set contains missing information of the field metadata type "missing field metadata of the ADS layer sales summary table and its upstream table," the metadata access paths corresponding to the ADS layer sales summary table and its upstream table are obtained from the initial entry layer and input into the metadata on-demand reading tool. The tool then obtains the detailed definition information of each field (such as amount, quantity, region, and date) corresponding to the ADS layer sales summary table and its upstream table, and assembles the complete field definition.

[0104] If it is determined that the missing information set contains missing information of bloodline relationship type, the data bloodline hierarchical traversal tool is invoked. The bloodline root node identifier in the current context and the preset traversal level depth are input into the data bloodline hierarchical traversal tool to obtain a hierarchical bloodline subgraph as the corresponding sub-detail information. Specifically, the data bloodline hierarchical traversal tool starts from the bloodline root node identifier and traces upstream or downstream dependency tables layer by layer according to the preset traversal level depth (such as traversing upwards by N levels of bloodline depth). It records the table name, the level to which it belongs, and the upstream and downstream dependencies between each level within the preset traversal level depth, and generates a hierarchical bloodline subgraph containing multi-level bloodline links accordingly. For example, if the missing information set contains the missing information of the lineage relationship type as "the complete multi-level lineage link of the ADS layer sales summary table", and the root node identifier in the current context is "the node identifier of the ADS layer sales summary table in the lineage relationship graph", and the preset traversal level depth is "traversing upwards by 2 levels of lineage depth", then input "the node identifier of the ADS layer sales summary table in the lineage relationship graph" and "traversing upwards by 2 levels of lineage depth" into the data lineage hierarchical traversal tool, and obtain a hierarchical lineage subgraph containing "DWD layer sales fact table → DWS layer sales detail table → ADS layer sales summary table" through the data lineage hierarchical traversal tool.

[0105] If the missing information set is determined to contain missing information of a computational logic type (such as "missing specific SQL calculation rules"), the SQL syntax logic parsing tool is invoked. The SQL template storage path in the current context is input into the SQL syntax logic parsing tool to obtain the SQL computational logic parsing result as the corresponding sub-detail information. Specifically, the SQL syntax logic parsing tool performs lexical analysis and syntax tree construction on the SQL statements in the SQL template storage path, extracting structured information such as the SELECT field list, FROM data source table, JOIN join conditions, WHERE filter conditions, GROUP BY grouping fields, and aggregate functions. Based on this structured information, it generates the SQL computational logic parsing result, which includes field mapping relationships, table join relationships, and a summary of computational logic.

[0106] If the missing information set is determined to contain missing information of the processing flow type (such as missing specific ETL processing logic), the ETL task fragment extraction tool is invoked. Based on the current context, several keywords of interest (such as "cleaning," "aggregation," "deduplication," etc.) are generated. The ETL script storage path and several keywords of interest in the current context are input into the ETL task fragment extraction tool to obtain a list of ETL core logic fragments as corresponding sub-detail information. Specifically, the ETL task fragment extraction tool reads the ETL script file in the ETL script storage path, segments the ETL script file according to the task execution stage, filters comment lines and log output statements, extracts the core processing logic fragment corresponding to each keyword of interest, and generates a list of ETL core logic fragments based on the extracted core processing logic fragments, the task execution stage to which each core processing logic fragment belongs, and the processing purpose.

[0107] In one embodiment, after performing the steps of obtaining a hierarchical lineage subgraph based on the root node identifier in the current context, a preset traversal level depth, and a preset data lineage hierarchical traversal tool, and using the hierarchical lineage subgraph as the corresponding sub-detail information, the coverage checking unit 13 is further configured to:

[0108] If the preset path analysis triggering conditions are determined based on the current context, then the preset data lineage analysis sub-agent is used to perform path analysis on the hierarchical lineage subgraph to obtain a lineage critical path analysis report.

[0109] Add the aforementioned critical path analysis report to the corresponding sub-detail information and update it.

[0110] In this embodiment, the information exploration strategy is further used to, after obtaining the corresponding sub-detail information by calling the tools in the active exploration tool layer, and determining, based on the current context, that the corresponding triggering condition is met, invoke the corresponding analysis sub-agent in the preset analysis sub-agent layer to execute the corresponding analysis task, thereby obtaining the corresponding analysis report to update the sub-detail information, so that the analysis report can be subsequently added to the current context. Specifically, after each analysis sub-agent completes its analysis, it only adds the conclusive analysis report to the corresponding sub-detail information, ensuring that intermediate reasoning processes are not injected into the current context, thus avoiding context pollution. By exploring the required sub-detail information through the tools in the active exploration tool layer and the analysis sub-agents in the analysis sub-agent layer, active information exploration according to a progressive disclosure approach is achieved.

[0111] Specifically, if the current context determines that the preset path analysis trigger conditions are met, indicating that lineage path analysis and risk assessment are required, then the data lineage analysis sub-agent in the analysis sub-agent layer further performs path analysis on the hierarchical lineage subgraph to obtain a lineage critical path analysis report. This report is then added to and updated in the corresponding sub-detail information to facilitate the subsequent addition of the hierarchical lineage subgraph and the lineage critical path analysis report to the current context. More specifically, lineage analysis concerns (such as locating the original data source of a certain indicator) are generated based on the current context, and the hierarchical lineage subgraph and the concerns are input into the data lineage analysis sub-agent in the analysis sub-agent layer for analysis. The data lineage analysis sub-agent, based on the concerns, identifies key dependency paths, branching nodes, and convergence nodes in the hierarchical lineage subgraph, assesses the data transmission integrity on each path, and generates a lineage critical path analysis report based on the identification and assessment results. The Critical Path Analysis Report includes the core dependency chain from the target data table to its source data table, descriptions of the data processing roles of each node, and potential risk points of lineage breakage.

[0112] In one embodiment, after executing the step of obtaining the SQL calculation logic parsing result based on the SQL template storage path in the current context and a preset SQL syntax logic parsing tool, and using the SQL calculation logic parsing result as the corresponding sub-detail information, the coverage checking unit 13 is further configured to:

[0113] If the preset logical translation triggering condition is determined based on the current context, the preset SQL logic analysis sub-agent is used to perform business semantic analysis on the SQL calculation logic parsing result to obtain an SQL business semantic analysis report.

[0114] Add the SQL business semantic analysis report to the corresponding sub-detail information and update it.

[0115] In this embodiment, if the preset logic translation trigger condition is met based on the current context, indicating that the SQL technical logic needs to be translated into business semantics, then the SQL logic analysis sub-agent in the analysis sub-agent layer further performs business semantic analysis on the SQL calculation logic parsing result to obtain an SQL business semantic analysis report. Then, the SQL business semantic analysis report is added to the corresponding sub-detail information and updated so that both the SQL calculation logic parsing result and the SQL business semantic analysis report can be subsequently added to the current context.

[0116] More specifically, based on the current context, SQL logic analysis concerns (such as understanding the calculation method of a certain indicator) are generated. The SQL calculation logic parsing results and SQL logic analysis concerns are input into the SQL logic analysis sub-agent in the analysis sub-agent layer for analysis and processing. Specifically, the SQL logic analysis sub-agent, based on the SQL logic analysis concerns, performs business semantic inference on the field mapping relationships, table relationships, and calculation logic summaries in the SQL calculation logic parsing results, translating the technical-level SQL logic into a business-level calculation rule description. Based on the translated calculation rule description, an SQL business semantic analysis report is generated. This report includes the business calculation method for each output field, explanations of business rules for data filtering and aggregation, and potential logical risks (such as implicit type conversions and omissions in null value handling).

[0117] In one embodiment, after executing the step of obtaining the ETL core logic fragment list based on the ETL script storage path in the current context and a preset ETL task fragment extraction tool, and using the ETL core logic fragment list as the corresponding sub-detail information, the coverage checking unit 13 is further configured to:

[0118] If the preset process analysis triggering conditions are determined to be met based on the current context, the preset ETL process analysis sub-agent is used to analyze the list of ETL core logic segments to obtain an ETL process analysis report.

[0119] Add the ETL process analysis report to the corresponding sub-detail information and update it.

[0120] In this embodiment, if the preset process analysis triggering conditions are met based on the current context, indicating that an integrity assessment and risk analysis of the ETL process is required, the ETL process analysis sub-agent in the sub-agent layer is further used to analyze the list of ETL core logic segments to obtain an ETL process analysis report. Then, the ETL process analysis report is added to the corresponding sub-detail information and updated, so that the list of ETL core logic segments and the ETL process analysis report can be subsequently added to the current context.

[0121] More specifically, based on the current context, ETL process analysis concerns (such as summarizing data cleaning rules) are generated. The list of core ETL logic segments and these concerns are then input into the ETL process analysis sub-agent in the analysis sub-agent layer for analysis. The ETL process analysis sub-agent, based on these concerns, reconstructs the execution order of the core ETL logic segments in the list, analyzes the data input sources, transformation rules, and output destinations at each processing stage, assesses the coverage of data quality control measures, and generates an ETL process analysis report. This report includes a description of the processing steps for each stage, a summary of data transformation rules, and potential data quality risks.

[0122] Context simplification element 14 is used to simplify the current context according to a preset context management strategy to obtain the minimum effective context if the check result is determined to be satisfactory.

[0123] In this embodiment, if the check result is determined to be satisfactory, it indicates that the current context meets the sufficiency conditions for analyzing the user's analysis needs. Then, according to a preset context management strategy, the current context is simplified to dynamically construct the minimum effective context. This ensures that the context is always the minimum effective set for completing the current analysis task, avoiding context overload and pollution, and improving the accuracy and response speed of analysis and reasoning. More specifically, the context management strategy uses the user's analysis needs as a benchmark to perform information filtering and granularity normalization on the current context. This eliminates irrelevant, redundant, and overlapping information, thereby removing information entries not directly related to the user's analysis needs (such as side nodes not on the core dependency path in lineage traversal), removing duplicate information, and replacing covered coarse-grained old information with finer-grained new information. After updating, the minimum effective context is obtained.

[0124] The analysis and reasoning unit 15 is used to perform analysis and reasoning based on the minimum effective context and output the analysis results.

[0125] In this embodiment, analysis and reasoning are performed based on the minimum effective context to complete the analysis and reasoning of user analysis requirements, thereby obtaining analysis results. This enables intelligent analysis of the big data warehouse, and the analysis results can assist data development and other related personnel in data development and analysis. Specifically, the analysis and reasoning unit 15 is used to: classify and integrate various types of information in the minimum effective context according to several analysis sub-tasks in the user analysis requirements, obtaining effective context information corresponding to each analysis sub-task; for each analysis sub-task, reasoning is performed on the analysis sub-task based on the corresponding effective context information to obtain analysis conclusions corresponding to each analysis sub-task; and the analysis conclusions corresponding to each analysis sub-task are summarized to form a complete analysis result. For example, for the data source category sub-task, the relevant lineage data in the minimum effective context is classified and integrated into the corresponding effective context information, and the data flow path is sorted out along the multi-level lineage links in the effective context information to generate source link descriptions, so as to obtain the analysis conclusions corresponding to the data source category sub-task. For ETL process subtasks, the relevant ETL processes and SQL rules in the minimum effective context are categorized and integrated into corresponding effective context information. Based on this effective context information, ETL processing steps and calculation methods are explained to obtain the analytical conclusions corresponding to this ETL process subtask. For risk identification subtasks, the relevant metadata constraints, ETL quality control measures, and SQL logic risk points in the minimum effective context are categorized and integrated into corresponding effective context information. Based on this effective context information, potential risk warnings are generated to obtain the analytical conclusions corresponding to this risk identification subtask.

[0126] The intelligent analysis device for big data warehouses provided in this invention is used to execute any of the aforementioned intelligent analysis methods for big data warehouses. By actively exploring data and dynamically constructing a minimum effective context, this invention significantly improves the accuracy and efficiency of big data warehouse analysis without relying on users to inject all information at once.

[0127] The aforementioned intelligent analysis method for big data warehouses can be implemented as a computer program, which can be used in, for example... Figure 3 It runs on the computer device shown.

[0128] Please see Figure 3 , Figure 3 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a device bus 501, wherein the memory may include a storage medium 503 and internal memory 504.

[0129] The storage medium 503 can store the operating system 5031 and the computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to execute intelligent analysis methods for big data warehouses.

[0130] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0131] The internal memory 504 provides an environment for the computer program 5032 in the storage medium 503 to run. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the intelligent analysis method for big data warehouse.

[0132] This network interface 505 is used for network communication, such as providing data transmission. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device 500 to which the present invention is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0133] The processor 502 is used to run the computer program 5032 stored in the memory to implement the big data warehouse intelligent analysis method disclosed in the embodiments of the present invention.

[0134] Those skilled in the art will understand that Figure 3 The embodiments of the computer device shown do not constitute a limitation on the specific configuration of the computer device. In other embodiments, the computer device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. For example, in some embodiments, the computer device may include only memory and a processor. In such embodiments, the structure and function of the memory and processor are different from those shown. Figure 3 The embodiments shown are consistent and will not be repeated here.

[0135] It should be understood that, in this embodiment of the invention, the processor 502 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0136] In another embodiment of the present invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the intelligent big data warehouse analysis method disclosed in the embodiments of the present invention.

[0137] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0138] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, systems, or units, or it may be an electrical, mechanical, or other form of connection.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0140] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a backend server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0142] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent analysis of big data warehouses, characterized in that, include: In response to user analysis requests, the system obtains target resource location information corresponding to the user analysis requests from a preset initial entry layer; wherein, the initial entry layer is used to store resource location information of the big data warehouse; the resource location information includes resource location address and summary description; The current context is constructed based on the user analysis requirements and the target resource location information; Based on a preset requirement coverage check strategy, check whether the current context meets the sufficiency conditions for analyzing the user analysis requirements, so as to obtain the check results. This includes: decomposing the user analysis requirements into several analysis sub-tasks; checking whether the current context contains the key information required to complete each analysis sub-task, so as to obtain the check results. If the inspection result is determined to be satisfactory, the current context is simplified according to a preset context management strategy to obtain the minimum effective context. Perform analytical reasoning based on the minimum effective context and output the analysis results; After checking whether the current context meets the sufficiency conditions for analyzing the user analysis requirements based on the preset requirement coverage check strategy, and obtaining the check result, the method further includes: If the inspection result is determined to be a non-satisfactory result, the corresponding data warehouse details are obtained according to the missing information set corresponding to the non-satisfactory result and the preset information exploration strategy. The current context and the data warehouse details are combined to form a correction context, and the correction context is simplified according to the context management strategy to obtain an optimized context; The steps involve updating the current context using the optimized context and returning to execute the preset requirement coverage check strategy to check whether the current context meets the sufficiency conditions for analyzing the user analysis requirements, so as to obtain the check results.

2. The intelligent analysis method for big data warehouses according to claim 1, characterized in that, The step of obtaining the corresponding data warehouse details based on the missing information set corresponding to the unsatisfactory results and a preset information exploration strategy includes: If it is determined that the missing information set contains missing information of the field metadata type, then the metadata access path corresponding to the missing information of the field metadata type is obtained from the initial entry layer, and the corresponding complete field definition is obtained based on the metadata access path and the preset metadata on-demand reading tool, and the complete field definition is used as the corresponding sub-detail information; If it is determined that the missing information set contains missing information of blood relationship type, then a hierarchical blood relationship subgraph is obtained based on the blood relationship root node identifier in the current context, the preset traversal level depth and the preset data blood relationship hierarchical traversal tool, and the hierarchical blood relationship subgraph is used as the corresponding sub-detail information; If it is determined that the missing information set contains missing information of the computational logic type, then the SQL computational logic parsing result is obtained based on the SQL template storage path in the current context and the preset SQL syntax logic parsing tool, and the SQL computational logic parsing result is used as the corresponding sub-detail information; If it is determined that the missing information set contains missing information of the processing flow type, then the ETL core logic fragment list is obtained based on the ETL script storage path in the current context and the preset ETL task fragment extraction tool, and the ETL core logic fragment list is used as the corresponding sub-detail information. All the obtained sub-detail information is combined to form the data warehouse detail information.

3. The intelligent analysis method for big data warehouses according to claim 2, characterized in that, After obtaining a hierarchical kinship subgraph based on the root node identifier in the current context, a preset traversal level depth, and a preset data kinship hierarchical traversal tool, and using the hierarchical kinship subgraph as the corresponding sub-detail information, the method further includes: If the preset path analysis triggering conditions are determined based on the current context, then the preset data lineage analysis sub-agent is used to perform path analysis on the hierarchical lineage subgraph to obtain a lineage critical path analysis report. Add the aforementioned critical path analysis report to the corresponding sub-detail information and update it.

4. The intelligent analysis method for big data warehouses according to claim 2, characterized in that, After obtaining the SQL calculation logic parsing result based on the SQL template storage path in the current context and the preset SQL syntax logic parsing tool, and using the SQL calculation logic parsing result as the corresponding sub-detail information, the method further includes: If the preset logical translation triggering conditions are met based on the current context, then the preset SQL logic analysis sub-agent is used to perform business semantic analysis on the SQL calculation logic parsing results to obtain an SQL business semantic analysis report. Add the SQL business semantic analysis report to the corresponding sub-detail information and update it.

5. The intelligent analysis method for big data warehouses according to claim 2, characterized in that, After obtaining the ETL core logic fragment list based on the ETL script storage path in the current context and a preset ETL task fragment extraction tool, and using the ETL core logic fragment list as the corresponding sub-detail information, the method further includes: If the preset process analysis triggering conditions are met based on the current context, the preset ETL process analysis sub-agent is used to analyze the list of ETL core logic segments to obtain an ETL process analysis report. Add the ETL process analysis report to the corresponding sub-detail information and update it.

6. The intelligent analysis method for big data warehouses according to claim 1, characterized in that, The step of obtaining target resource location information corresponding to the user's analysis needs from a preset initial entry layer includes: Semantic parsing is performed on the user analysis requirements to obtain the target data table name; Obtain target resource location information related to the target data table name from the initial entry layer; wherein, the target resource location information includes data warehouse hierarchical topology information, core table metadata index information, data lineage root node information, ETL task overview information, and business SQL template index information related to the target data table name.

7. A big data warehouse intelligent analysis device, characterized in that, include: A demand response unit is used to respond to user analysis demands and obtain target resource location information corresponding to the user analysis demand from a preset initial entry layer; wherein, the initial entry layer is used to store resource location information of the big data warehouse; the resource location information includes resource location address and summary description; A context building unit is used to build the current context based on the user analysis requirements and the target resource location information. The coverage checking unit is used to check whether the current context meets the sufficiency conditions for analyzing the user analysis requirements based on a preset requirement coverage checking strategy, so as to obtain the checking result. The check includes: decomposing the user analysis requirements into several analysis sub-tasks; checking whether the current context contains the key information required to complete each analysis sub-task, so as to obtain the checking result. The context simplification element is used to simplify the current context according to a preset context management strategy to obtain the minimum effective context if the check result is determined to be a satisfactory result. An analysis and reasoning unit is used to perform analysis and reasoning based on the minimum effective context and output the analysis results; After executing the step of checking whether the current context meets the sufficiency conditions for analyzing the user analysis requirements based on the preset requirement coverage check strategy, and obtaining the check result, the coverage check unit is further configured to: If the inspection result is determined to be a non-satisfactory result, the corresponding data warehouse details are obtained according to the missing information set corresponding to the non-satisfactory result and the preset information exploration strategy. The current context and the data warehouse details are combined to form a correction context, and the correction context is simplified according to the context management strategy to obtain an optimized context; The steps involve updating the current context using the optimized context and returning to execute the preset requirement coverage check strategy to check whether the current context meets the sufficiency conditions for analyzing the user analysis requirements, so as to obtain the check results.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the big data warehouse intelligent analysis method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the intelligent analysis method for big data warehouses as described in any one of claims 1 to 6.

Images