A customizable automated data model analysis tool and methods of use thereof

Abstract

The application provides a customized automatic data model analysis tool and a use method thereof, and belongs to the technical field of data storage, and specifically comprises a configuration module, an analysis metadata module and an output module; the configuration module is used for reading a data source file, and automatically configuring parameters according to the type of the data source file; the analysis metadata module automatically extracts metadata structure information of a data source and table data of the data source according to different types of data source file models, and maps field information of the metadata structure information and each row value stored by the field information of the metadata structure information, so as to realize that a user establishes a customized analysis model; and the output module is used for calling a data processing algorithm to complete data filling of the model according to the established analysis model, and further realizing information display according to business requirements. Through the processing scheme, structured data sources and unstructured data sources are uniformly analyzed, and different dimension data models are set according to user requirements.

Description

Technical Field

[0001] This application relates to the field of data storage, and in particular to a customizable automated data model analysis tool and its usage. Background Technology

[0002] Traditional data application development requires thorough communication with users to define the required fields, data types, and retrieval methods, and typically uses a relational database for data storage. This approach is suitable for applications with fixed requirements, infrequent business rule changes, and users with basic data manipulation skills. However, for current applications with frequently changing requirements, large data volumes, and complex data sources, this approach is inflexible, inefficient, and limited in its ability to utilize other data sources. With the deepening integration of digital technology and industrialization, the utilization of document-based and non-relational data, as well as the fusion and analysis of data with relational databases, has become increasingly important. A purely relational database-based data model is no longer adequate for today's rapidly evolving business needs. Summary of the Invention

[0003] In view of this, this application provides a customizable automated data model analysis tool and its usage method, which solves the problems in the prior art, provides unified analysis for structured and unstructured data sources, and sets different dimensional data models according to user needs.

[0004] The customizable automated data model analysis tool provided in this application adopts the following technical solution:

[0005] A customizable automated data model analysis tool, characterized by including a configuration module, an analysis metadata module, and an output module;

[0006] The configuration module is used to read the data source file and automatically configure parameters according to the type of the data source file.

[0007] The metadata analysis module automatically extracts metadata structure information and table data of the data source for different types of data source file models, and maps the field information of the metadata structure information to each row value stored in the metadata structure information, so as to enable users to build customized analysis models.

[0008] The output module is used to call data processing algorithms to populate the data of the established analysis model and further display information according to business needs.

[0009] Optionally, the analysis tool includes a unified API interface for analyzing and extracting the data source structure, storing the metadata structure information, and calling the corresponding analysis program through the API interface based on the configured data source parameters.

[0010] Optionally, the analysis metadata module constructs a JSON-formatted character data as a unified data exchange format for source data and analysis data.

[0011] Optionally, the analysis metadata module uses a JSON structure to create metadata structure information and table data of the data source, and maps the field information of the metadata structure information to each row value stored in the field information of the metadata structure information, so as to enable users to build customized analysis models.

[0012] Optionally, the metadata analysis module analyzes the metadata structure information of the data table in the data source and stores it as a set of feature dimensions for users to select and configure when using it.

[0013] Optionally, the data source file is defined as structured data source file information and unstructured data source file information. Structured data file information includes database connection parameters, database table names, view names, query field names, query conditions, grouping conditions, and sorting rules, etc. Unstructured data source information includes source file path address, file name, file extension, and file format.

[0014] Optionally, the configuration module reads the table names, field names, field types, and data of the database of the data source.

[0015] On the other hand, the method of using the customizable automated data model analysis tool provided in this application adopts the following technical solution:

[0016] A method for using a customizable automated data modeling tool, including:

[0017] Step 1: Launch the customizable automated data modeling tool as described above;

[0018] Step 2: Configure a timed task to execute a background program that reads the source file.

[0019] Step 3: Construct a JSON-formatted string to store the field type information in the data source metadata structure information, forming the metadata management information of the original table. The metadata management information includes field names and field types.

[0020] Step 4: Construct each row of data in the data management information into a JSON-formatted string based on the format of the field type information;

[0021] Step 5: Combine each JSON string to form a JSON array object containing the entire table's data;

[0022] Step 6: Based on the metadata management information, select the analysis dimension fields and fact data fields, generate the dimension table metadata information and fact table metadata information respectively, and build the application analysis data model;

[0023] Step 7: Define the business rules for the values ​​in the fact table metadata fields and the original data fields, according to the business rules.

[0024] Step 8: Convert the JSON string into a hash set.<key,value> Format object,

[0025] Step 9: Using a recursive algorithm, call the operation in Step 7, and for each field name obtained by looping through the set of field names in Step 3, take the value object in the Hash set as the field value of that field name, calculate and store it under the new field name, and recursively until the JSON array is completely traversed.

[0026] Step 10: Call the persistence program to select different storage media to save the fact table data and dimension table data.

[0027] In summary, this application includes the following beneficial technical effects:

[0028] 1. Define different types of data sources through parameter configuration, including relational data sources and document data sources. Call a unified API interface to analyze the data source structure and store metadata information according to different data source formats.

[0029] 2. Convert data from different data sources into JSON format and save the data.

[0030] 3. By analyzing JSON format metadata, the attribute columns of the data source are automatically parsed out for users to choose freely, thereby achieving customizable goals without having to collect user requirements or predefine the analytical database table structure.

[0031] 4. A recursive algorithm was designed, which uses the divide-and-conquer approach to decompose large table data into a set of smaller data sets, thereby improving program efficiency and reducing the pressure on server and terminal data I / O.

[0032] 5. A modeling method is provided based on business requirements, which realizes the data conversion between business model and analysis model. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating the usage method of the customizable automated data model analysis tool for this application;

[0035] Figure 2 This is the original data source model for the embodiments of this application;

[0036] Figure 3 This is the dimension table model of an embodiment of this application. Detailed Implementation

[0037] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0038] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0040] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0041] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0042] This application provides a customizable automated data model analysis tool.

[0043] A customizable automated data model analysis tool, including a configuration module, an analysis metadata module, and an output module.

[0044] The configuration module is used to read the data source file and automatically configure parameters according to the type of the data source file; the configuration module can automatically configure structured data sources and unstructured data sources according to the user's parameters.

[0045] The data source file is defined as both structured and unstructured data source file information. Structured data file information includes database connection parameters, database table names, view names, query field names, query conditions, grouping conditions, and sorting rules, etc. Unstructured data source information includes source file path address, file name, file extension, and file format.

[0046] The configuration module reads the table names, field names, field types, and data of the database in the data source.

[0047] The metadata analysis module automatically extracts metadata structure information and table data of the data source for different types of data source file models, and maps the field information of the metadata structure information to each row value stored in the metadata structure information, so as to enable users to build customized analysis models.

[0048] Specifically, the metadata analysis module constructs a JSON-formatted character data as a unified data exchange format for source and analysis data. It uses the JSON structure to create the metadata structure information and table data of the data source, mapping the field information of the metadata structure information to each row value stored in the metadata structure information, enabling users to build customized analysis models. The metadata analysis module analyzes the metadata structure information of the data source's tables and stores it as a set of feature dimensions for users to select and configure during use.

[0049] The output module is used to call data processing algorithms to populate the data of the established analysis model and further display information according to business needs.

[0050] The analysis tool uses a unified API interface to analyze and extract the data source structure, store the metadata structure information, and call the corresponding analysis program through the API interface based on the configured data source parameters.

[0051] like Figure 1 As shown, this application also discloses a method for using a customizable automated data modeling tool, including:

[0052] Step 1: Launch the customizable automated data model tool.

[0053] Step 2: Configure a timed task to execute a background program that reads the source file.

[0054] Step 3: Construct a JSON-formatted string to store the field type information in the data source table structure, forming the metadata management information of the original table. The metadata management information includes field names and field types, in the format (field name 1: field type 1, field name 2: field type 2, ... field name n: field type n).

[0055] Step 4: Construct each row of data in the database of data management information into a JSON format string according to the format of the field type information; that is, construct each row of data in the database table into a JSON format string according to the format of (field name 1: field value 1, field name 2: field value 2, ..., field name n: field value n).

[0056] Step 5: Combine each JSON string to form a JSON array object containing the entire table's data;

[0057] Step Six: Based on the metadata management information, select the analysis dimension fields and fact data fields, generate the dimension table metadata information and fact table metadata information respectively, and build the application analysis data model;

[0058] Step 7: Define the business rules for the values ​​in the fact table metadata fields and the original data fields, according to the business rules.

[0059] Step 8: Convert the JSON string into a hash set<key,value> Format object,

[0060] Step 9: Using a recursive algorithm, call the operation in Step 7, and take the set of field names {field name 1, field name 2, ... field name n} from Step 3. For each field name obtained by looping through, take the value object in the Hash set as the field value of that field name, calculate and store it under the new field name, and recursively until the JSON array is completely traversed.

[0061] Step 10: Call the persistence program to select different storage media to save the fact table data and dimension table data.

[0062] The analytical tool in this application allows users to build their own business analysis models based on their needs, enabling the calculation and mapping of data from operational business data to analytical factual data. This provides a possibility for users to self-implement their business analysis requirements.

[0063] Specifically, this application provides examples for both structured data sources and non-institutional data sources.

[0064] 1. Structured database tables:

[0065] Data source JDBC: com.microsoft.sqlserver.jdbc.SQLServerDriver.

[0066] Data source URL: jdbc:sqlserver: / / XX:XX; DatabaseName=bpmpridb.

[0067] Data source username and password.

[0068] 2. Database table structure information to be read:

[0069] Table: Product Information Table {

[0070] Product model number varchar(30),

[0071] Product name varchar(50), ...

[0073] }

[0074] table Production Task Table {

[0075] Project code varchar(30),

[0076] Product Name varchar(30),

[0077] Delivery quantity

[0078] Delivery Date ...

[0080] } ...

[0082] 3. Information from unstructured data tables:

[0083] Excel project plan.

[0084] Field name (topic code, project name, ...).

[0085] 4. The original data source model is as follows: Figure 2 As shown, this is a relational data model.

[0086] 5. If it is 2, retrieve the component table information from the structured data source using the getString("Table_Name component") method of the DatabaseMetaData class.

[0087] 6. Use the getColumns method of the DatabaseMetaData class to iterate through the table structure fields.

[0088] 7. Define the field names and field types of the data source table structure as a JSON format string: {"Product Model":varchar(30),"Product Name":varchar(30),...}.

[0089] 8. If it is step 3, retrieve the table object through the HSSFWorkbook class, retrieve the field name and table field type of each column, and then repeat steps 6 and 7.

[0090] 9. Extract metadata information from the JSON string generated in step 7 to allow users to create dimension tables and fact tables.

[0091] 10. For the row set data obtained from the query, construct a JSON-formatted string {<product model: n1>, <product name: v1>, ...}.

[0092] 11. Convert the JSON-formatted string to a JSON object, and convert the entire table of data into a JSON array object [{<product model: n1>, <product name: v1>, ...}, {<product model: n2>, <product name: v2>, ...}].

[0093] 14. Construct a custom data storage array based on JSON, using the collection list type: [{"column1":"n1","column2":"v1"...},{"column1":"n2","column2":"v2"...}...{"column1":"nN","column2":"vN",...}].

[0094] 15. Use a recursive algorithm to traverse the JSON array and convert the obtained JSON objects into a HashMap.<key,value> Based on user requirements, the data field name is set to columnT, and the value corresponding to this column is obtained through HashMap mapping.

[0095] 16. Generate fact table and dimension table models, such as Figure 3 As shown.

[0096] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A customizable automated data model analysis tool, characterized in that, It includes a configuration module, an analysis metadata module, and an output module; The configuration module is used to read the data source file and automatically configure parameters according to the type of the data source file. The metadata analysis module automatically extracts metadata structure information and table data of the data source for different types of data source file models, and maps the field information of the metadata structure information to each row value stored in the field information of the metadata structure information, so as to enable users to build customized analysis models. The output module is used to call data processing algorithms to populate the data of the established analysis model and display information according to business needs.

2. The customizable automated data model analysis tool according to claim 1, characterized in that, The analysis tool uses a unified API interface to analyze and extract the data source structure, store the metadata structure information, and call the corresponding analysis program through the API interface based on the configured data source parameters.

3. The customizable automated data model analysis tool according to claim 1, characterized in that, The analysis metadata module constructs a JSON-formatted character data as a unified data exchange format for source and analysis data.

4. The customizable automated data model analysis tool according to claim 3, characterized in that, The analysis metadata module uses a JSON structure to create metadata structure information and table data for the data source. It maps the field information of the metadata structure information to the value of each row of the field information stored in the metadata structure information, enabling users to build customized analysis models.

5. The customizable automated data model analysis tool according to claim 3, characterized in that, The metadata analysis module analyzes the metadata structure information of the data table in the data source and stores it as a set of feature dimensions for users to select and configure when using it.

6. The customizable automated data model analysis tool according to any one of claims 1-5, characterized in that, The data source file is defined as both structured and unstructured data source file information. Structured data source file information includes database connection parameters, database table names, view names, query field names, query conditions, grouping conditions, and sorting rules. Unstructured data source information includes source file path address, file name, file extension, and file format.

7. The customizable automated data model analysis tool according to claim 6, characterized in that, The configuration module reads the table names, field names, field types, and data of the database in the data source.

8. A method for using a customizable automated data model analysis tool, characterized in that, include: Step 1: Launch the customizable automated data model analysis tool as described in any one of claims 1-7; Step 2: Configure a timed task to execute a background program that reads the source file. Step 3: Construct a JSON-formatted string to store the field type information in the data source metadata structure information, forming the metadata management information of the original table. The metadata management information includes field names and field types. Step 4: Construct each row of data in the data management information into a JSON-formatted string based on the format of the field type information; Step 5: Combine each JSON string to form a JSON array object containing the entire table's data; Step 6: Based on the metadata management information, select the analysis dimension fields and fact data fields, generate the dimension table metadata information and fact table metadata information respectively, and build the application analysis data model; Step 7: Define the business rules for the values ​​in the fact table metadata fields and the original data fields, according to the business rules. Step 8: Convert the JSON string into a hash set.<key,value> Format object; Step 9: Using a recursive algorithm, call the operation in Step 7, and take the field name set from Step 3. For each field name obtained by looping through it, take the value object in the Hash set as the field value of that field name, calculate and store it under the new field name, and recursively until the JSON array is completely traversed. Step 10: Call the persistence program to select different storage media to save the fact table data and dimension table data.

Images