Self-adaptive file output system based on multi-way tree model number bin
By adopting the multi -bar tree model and adaptive file output system in the data warehouse, dynamically adjust the data inquiry path and output file format, solve the problem of insufficient flexibility when dealing with complex queries and frequent changes in business needs, and realize Efficient and flexible data processing and output mechanisms.
Patent Information
- Application Number
- CN202411385591.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-01-21
AI Technical Summary
When existing data warehouses deal with complex queries and frequently changing business needs, fixed query paths and output file formats lead to insufficient system flexibility and difficult to meet diversified business needs.
Adopt the adaptive file output system based on the multi -fork tree model, build a module, query the construction module, data extraction module, and output control module through the position of the position, dynamically adjust the data query path and output file format, and generate different types of output according to business needs document.
It improves the flexibility and query efficiency of the system, can dynamically adapt to the changes in business needs, reduce the frequency of system updates, and generate work efficiency and data quality through automated output files.
Smart Images
Figure CN119336791A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a self-adaptive file output system based on a multi-tree model data warehouse. Background Art
[0002] A large amount of business data is generated during the operation of an enterprise. It is usually captured and stored in a relational database designed for specific transactions. Different databases are associated with each other according to business logic. At the same time, a large number of databases are managed and analyzed in the form of data warehouses. There are a large number of query and statistical requirements in the data warehouse, and data needs to be extracted from different databases for combination and analysis.
[0003] For performance requirements and easy management, current data warehouses mostly use fixed query paths and query entries to ensure data consistency and accuracy. However, this approach may limit the flexibility of the system, making some complex queries difficult to implement. If business requirements change, fixed query paths may not be able to adapt to new requirements. If requirements change frequently, the system needs to be updated frequently.
[0004] At the same time, different query services correspond to output file requirements with different contents and formats. Due to the lack of a unified output file control method, users need to perform editing and format conversion steps after obtaining the query data, which increases the workload and affects work efficiency.
[0005] Therefore, it is necessary to provide a flexible and efficient data processing and output mechanism to meet diverse business needs. Summary of the invention
[0006] The present invention provides a self-adaptive file output system based on a multi-tree model data warehouse, which can dynamically adjust data queries according to actual needs, while increasing the control and management of output files, and output different files according to business needs.
[0007] This application provides the following technical solutions:
[0008] The self-adaptive file output system based on the multi-tree model data warehouse includes a data warehouse construction module, a query construction module, a data extraction module and an output control module, wherein:
[0009] A data warehouse construction module is used to construct a data warehouse based on a multi-tree model. The data warehouse contains one or more data models, each of which contains multiple database tables organized according to the multi-tree model. The root node and leaf nodes of the multi-tree model are data nodes containing database tables. The root node is the query entry of the data model. The connection between the root node and the leaf nodes and between the leaf nodes is the conditional query relationship between the corresponding database tables.
[0010] A query construction module, used to construct a data query link according to business requirements and a multi-branch tree model, wherein the business requirements include data node information storing target data;
[0011] A data extraction module, used to extract data from the data model according to the data query link, and also used to perform data processing operations on the extracted data;
[0012] The output control module is used to integrate the processed data and generate output files that are compatible with the business requirement type.
[0013] Furthermore, any leaf node in the established multi-branch tree model can be defined as a new root node, and the original root node can be defined as a new leaf node.
[0014] Furthermore, the multi-branch tree model is regenerated when any leaf node is defined as a new root node.
[0015] Furthermore, the connections in the multi-branch tree model contain weight information, and the query construction module generates a data query link according to the weight information and data node information of business requirements.
[0016] Furthermore, the query construction module constructs multiple query links in parallel, and the data extraction module uniformly executes all query links generated by the query construction module.
[0017] Furthermore, the data extraction module includes an extraction logic configuration unit, a data extraction unit and a data processing unit, wherein the extraction logic configuration unit is used to define the data extraction logic; the data extraction unit extracts data according to the data extraction logic and the data query link; and the data processing unit completes data processing on the extracted data.
[0018] Furthermore, the data processing includes data cleaning, conversion and formatting operations.
[0019] Further, the output control module includes an output file configuration unit and an output file generation unit, wherein:
[0020] The output file configuration unit is used to configure the parameters and configuration items of the output file, the parameters include output fields, field name mapping, content conversion rules, and the configuration items include file type, file format, encoding method, file path and naming rules;
[0021] The output file generation unit is used to read the configured parameters and configuration items, integrate the processed data, and generate an output file.
[0022] The principle and advantage of the present invention are as follows: In the present application, the data warehouse is organized according to multiple data models including a multi-branch tree model, and the logic of the conditional query relationship between database tables is continuously implemented downward through the root node. The data warehouse automatically matches each tree node in the data model for data query based on the data in the designated root node database table and the data in the database table to be queried. Since the leaf nodes and the root nodes are designed as information that can be converted to each other, the entire data warehouse can form a network structure, and a multi-branch tree model is dynamically generated after each node attribute update, which can enhance the flexibility of the system, and can select a closer query entry according to actual query needs to improve query efficiency; at the same time, it can also better adapt to changes in demand.
[0023] When building a query link, the optimal path is calculated based on the connection weight, and a parallel construction method is adopted to quickly obtain query links for multiple different query services. Compared with the serial method of build-execute-build, the query efficiency of the system can be further improved.
[0024] At the same time, an output control module is designed in this application to manage output files, which can automatically complete data integration and data processing for query data, and generate standardized output files according to business types, which can help system administrators avoid tedious operations of data conversion and file editing, and focus more on the design of business logic and data analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a structural diagram of the self-adaptive file output system based on the multi-tree model data warehouse of the present invention;
[0026] Figure 2 It is a schematic diagram of the association relationship between data nodes in the data warehouse;
[0027] Figure 3 This is a schematic diagram of the results of building a data query link in the data warehouse. DETAILED DESCRIPTION
[0028] Each data node in the data warehouse usually contains one or more databases designed for specific transactions, storing various business data generated online. Different business departments or different users need to extract specific data from the data warehouse for analysis. For example, the finance department needs to analyze product sales, business operations, cost management, etc., while the human resources department needs to analyze personnel salaries, performance appraisals, attendance information, etc.; the operation and maintenance department needs to analyze the operating performance and stability of a certain business system. The present invention provides a self-adaptive file output system based on a multi-branch tree model data warehouse, which realizes the automation of the entire process from data query to file output.
[0029] The following is further described in detail through specific implementation methods:
[0030] Embodiment 1
[0031] This embodiment is a self-adaptive file output system based on a multi-tree model data warehouse. Figure 1 As shown, it includes a data warehouse construction module, a query construction module, a data extraction module and an output control module, wherein:
[0032] The data warehouse construction module is used to build a data warehouse based on a multi-branch tree model. The data warehouse contains one or more data models, each of which contains multiple database tables organized according to the multi-branch tree model. The multi-branch tree model contains root nodes, leaf nodes, and connections between nodes. Among them, the root node and the leaf node are data nodes containing database tables, the root node is the query entry of the data model, and the connection between the nodes represents the conditional query relationship between the corresponding database tables. Starting from the root node, traverse downward to the leaf nodes that have an association relationship with the root node, and then start traversing other next-level nodes with an association relationship from each leaf node with an association relationship. Repeat this process until all data nodes in the data warehouse are traversed to complete the construction of the multi-branch tree model. In the process of building the multi-branch tree model, weight information is configured for each connection between nodes.
[0033] For example, for Figure 2 The database association diagram shown in the figure constructs a data model, which contains nodes A, B, C, D, E, F, G, H, I, J, X1~X8. The gray arrows between the nodes represent the conditional query relationship. Figure 2 It can be seen that if node A is used as the root node, the diagram can be divided into two multi-branch tree models. The first multi-branch tree contains nodes B, C, D, E, X1~X4, and the second multi-branch tree contains nodes F, G, H, I, J, X5~X8. If node C is used as the root node, the data model contains a multi-branch tree model, which contains all data nodes.
[0034] Any leaf node in the established multi-branch tree model can be defined as a new root node, and the original root node can be defined as a new leaf node. In other words, the query entry of the multi-branch tree model can be redefined, and when any leaf node is defined as a new root node, the multi-branch tree model will be regenerated starting from the new root node. Different root node selections may affect the length and latency of the specific query path. Reasonable selection of the root node can achieve dynamic optimization of the data query path.
[0035] The query construction module constructs a data query link based on the data nodes and multi-tree model of business requirements. After the data model is constructed, a corresponding data query link needs to be constructed for each query requirement. To construct a data query link, first, it is necessary to determine the data nodes that need to be executed according to the query requirements, that is, it is necessary to access the target node in the multi-tree model, and then start from each target node, query the data node with an association relationship with the target node to the upper level, and start from the data node to query the data node with an association relationship at a higher level again, and repeat this process until the query ends at the root node. The target data node, all intermediate nodes, the root node, and the lines between all intermediate nodes constitute a data query link. According to this process, multiple data query links may be constructed from a certain target node to the root node, and the optimal path is selected by summing the weights of the lines between the intermediate nodes. The data query links obtained from different target nodes may partially overlap. At this time, the data links with partial overlap can be merged, and the longest path is used as the unified query link for all target nodes on the link.
[0036] For example, the target nodes for this query are determined to be A, D, I, and J, and the associated nodes are queried upward from the target data node. Figure 2 For example, if the root node of the data model is node A, the system starts from node D, and is associated upward to node C through the conditional query relationship. Node C is then associated upward to node B, and node B is finally associated upward to node A, and finally a query link A->B->C->D is obtained. This link can realize the query of nodes A and D starting from the query entry. For node I, the query link A->H->I can be obtained in the same way. When node J queries upward, it is associated with node I. Since node I has already established a query link, it is directly added to the link to obtain A->H->I->J; at the same time, another link A->F->G->I->J can be constructed. The links A->H->I->J and A->F->G->I->J can both realize access to nodes I and J. By calculating the weights of the two links, A->H->I->J is selected as the best. Finally, two query links are constructed: A->B->C->D and A->H->I->J. Figure 3 shown.
[0037] If the root node of the data model is selected as node B, according to the above process, two different query links are finally constructed: B->C->D and B->A->H->I->J.
[0038] Comparing the above two situations, it can be seen that when node A is selected as the root node, the path length of the query link is more balanced and the query speed is better in actual execution.
[0039] The data extraction module includes an extraction logic configuration unit, a data extraction unit and a data processing unit, wherein the extraction logic configuration unit is used to configure the data extraction logic, including using SQL statements, calling APIs or other methods; after the query construction module constructs a query link for all target data nodes, the data extraction unit uniformly executes the constructed data query link, accesses the target data nodes to extract data according to the method defined by the data extraction logic, and constructs each queried data node into a combined data node according to the fields to be output. The data processing unit completes the data processing tasks for the extracted data, including data cleaning, conversion and formatting operations, to ensure the consistency and quality of the data.
[0040] The processed data is integrated by the output control module, and an output file that is compatible with the type of business requirements is generated. The output control module includes an output file configuration unit and an output file generation unit. Among them, the output file configuration unit is used to configure the parameters and configuration items of the output file. The parameters of the output file include output fields, field name mapping, and content conversion rules. Field mapping is to replace the field name and configuration item of the combined data node with the field name required by the business; content conversion is to convert the value of the field into a format, such as converting the timestamp into a unified date format, unifying the decimal places of values in different databases, etc. The configuration items of the output file include file type, file format, encoding method, file path, and naming rules. The output file generation unit is used to read the configured parameters and configuration items, integrate the processed data, and generate an output file in a specific format, such as CSV, Excel, or JSON.
[0041] For example, the human resources department needs to generate the comprehensive evaluation results of individual employees in Excel format. First, it needs to determine the target data nodes to be accessed according to the query requirements. These target data nodes store data tables such as organizational structure, salary information, attendance information, performance statistics, KPI data, etc. The query entry is determined based on the information of the data node. The system dynamically generates a multi-branch tree model based on the query entry and the information of the target data node, and selects the optimal path according to the weight calculation to build multiple data query links. Furthermore, the system reads the pre-configured data extraction logic according to the type of comprehensive evaluation query business, executes the data extraction logic, extracts data in the specified fields and specified ranges from each data table, and performs data cleaning, data conversion and formatting operations on the extracted data to ensure data consistency. After completing the data processing, the data is grouped according to the scope of departments, project groups, and employees. According to the evaluation rules, multi-level evaluation results such as department level, project group level, and employee level are generated respectively. The field names and data ranges included in the evaluation result tables at different levels are different. After the evaluation results are generated, the system writes the evaluation results of different levels into the corresponding Excel format output files according to the configuration of the output control module, names the output files according to the query business and evaluation level definition, and outputs them to the corresponding file directory. For example, the output file name of the overall evaluation results of the software department in 2023 is "Software Department 2023 Evaluation Results", which contains the software department's salary settings, department KPI settings, department KPI completion data, performance adjustment coefficients, department evaluation results and other data. The structure of the output directory is set to "Human Resources Department-Department Comprehensive Evaluation-2023"; and the output file name of the 2023 evaluation results of an employee of the software department is "xxx2023 Annual Evaluation Results", which contains the employee's salary settings, KPI settings, KPI completion data, attendance information, performance adjustment coefficients, personal evaluation results and other data. The structure of the output directory is set to "Human Resources Department-Department Comprehensive Evaluation-2023". In this way, a fully automated process from data query to file output is realized.
[0042] The above are only embodiments of the present invention. The invention is not limited to the field involved in this implementation case. The common knowledge such as the known specific structure and characteristics in the scheme is not described in detail here. It should be pointed out that for those skilled in the art, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.
Claims
1. The self-adaptive file output system based on the multi-tree model data warehouse is characterized by: It includes data warehouse construction module, query construction module, data extraction module and output control module, among which: A data warehouse construction module is used to construct a data warehouse based on a multi-tree model. The data warehouse contains one or more data models, each of which contains multiple database tables organized according to the multi-tree model. The root node and leaf nodes of the multi-tree model are data nodes containing database tables. The root node is the query entry of the data model. The connection between the root node and the leaf nodes and between the leaf nodes is the conditional query relationship between the corresponding database tables. A query construction module, used to construct a data query link according to business requirements and a multi-branch tree model, wherein the business requirements include data node information storing target data; A data extraction module, used to extract data from the data model according to the data query link, and also used to perform data processing operations on the extracted data; The output control module is used to integrate the processed data and generate output files of the type that matches the business requirements.
2. The self-adaptive file output system based on multi-tree model data warehouse according to claim 1 is characterized in that: Any leaf node in the constructed multi-branch tree model can be defined as a new root node, and the original root node can be defined as a new leaf node.
3. The self-adaptive file output system based on multi-tree model data warehouse according to claim 2 is characterized in that: The multi-branch tree model is regenerated when any leaf node is defined as a new root node.
4. The self-adaptive file output system based on multi-tree model data warehouse according to claim 1, characterized in that: The connections in the multi-branch tree model contain weight information, and the query construction module generates a data query link according to the weight information and data node information of business requirements.
5. The self-adaptive file output system based on multi-tree model data warehouse according to claim 1, characterized in that: The query construction module constructs multiple query links in parallel, and the data extraction module uniformly executes all query links generated by the query construction module.
6. The self-adaptive file output system based on multi-tree model data warehouse according to claim 1, characterized in that: The data extraction module includes an extraction logic configuration unit, a data extraction unit and a data processing unit. The extraction logic configuration unit is used to define the data extraction logic; the data extraction unit extracts data according to the data extraction logic and the data query link; and the data processing unit completes data processing on the extracted data.
7. The self-adaptive file output system based on multi-tree model data warehouse according to claim 1, characterized in that: The data processing includes data cleaning, conversion and formatting operations.
8. The self-adaptive file output system based on multi-tree model data warehouse according to claim 1, characterized in that: The output control module includes an output file configuration unit and an output file generation unit, wherein: The output file configuration unit is used to configure the parameters and configuration items of the output file, the parameters include output fields, field name mapping, content conversion rules, and the configuration items include file type, file format, encoding method, file path and naming rules; The output file generation unit is used to read the configured parameters and configuration items, integrate the processed data, and generate an output file.