Visual generation self-service access method and system based on business metadata semantics

Abstract

The invention relates to the technical field of databases, and discloses a service metadata semanteme-based visual generation self-service access method and system, and the method comprises the steps: S1, determining a first target relational data source, a data source type and a physical script grammar; s2, storing the business metadata and the first mapping relation, and obtaining target business metadata; s3, generating a service data set, wherein the service data set comprises a first logic script; s4, quoting the business data set, and generating a second logic script as an access task; s5, analyzing the first logic script to generate a first physical script, and analyzing the second logic script to generate a second physical script; s6, splicing the first physical script and the second physical script to obtain a third physical script; and S7, in response to successful execution of the third physical script, downloading target data corresponding to the data access task. The method has the effect of reducing the data access threshold of the database.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a self-service data retrieval method and system based on visualization of business metadata semantics. Background Art

[0002] With the continuous growth of enterprise data volume and the increasing complexity of business needs, the existing method of retrieving data based on technical metadata and directly connecting to the database by writing SQL codes can no longer meet the needs of business personnel for fast and flexible data acquisition.

[0003] The threshold for obtaining data by writing SQL codes is relatively high. For business personnel, writing SQL codes requires certain database knowledge and programming skills. For technical personnel, they also need to constantly learn and master new SQL syntax and functions to adapt to the ever-changing database system and business needs. Secondly, when writing SQL codes, syntax errors, logical errors and other problems are prone to occur, resulting in inaccurate query results or failure to obtain the required data. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for self-service data retrieval based on business metadata semantics, which can realize the conversion of data retrieval tasks into corresponding logic scripts according to the user's graphical instructions, the mapping relationship between business metadata and technical metadata, and the mapping relationship between technical metadata and entity objects. According to the type of database, the appropriate syntax is matched to the parsed logic script, and a physical script for data retrieval is generated to execute the data retrieval function in the specified database. This method has the effect of lowering the threshold for database data retrieval, without the need to edit SQL statements when business personnel retrieve data from the database.

[0005] The present invention is achieved through the following technical solutions: In a first aspect, the present application provides a self-service data retrieval method based on business metadata semantic visualization, comprising the following steps: Step S1: according to the data source definition instruction from the target user, determine the first target relational data source and obtain the data source type of each data source and the physical script syntax corresponding to the data source type; Through the above technical solution, it is possible to select a relational data source from among a variety of connectable data sources as a data source for alternative selection according to the user's instructions, that is, the first target relational data source, the data source type includes a first type classified according to the major database categories, and a second type of specific models derived from the first type, such as the first type may include MySQL, PostgreSQL and other categories, and a suitable physical script syntax may be matched for each data source according to the first type, or a suitable physical script syntax may be further matched for each data source according to the second type; It can shield the differences between the underlying databases to further provide a unified query engine layer for different types of databases, thereby quickly adapting to multiple relational databases and flexibly responding to various business needs; The target user's data source definition instructions can be visual instructions, such as drag / click / input definition, etc.

[0006] Step S2: according to the metadata definition instruction from the target user, save the business metadata defined according to the business direction and save the first mapping relationship between the technical metadata and the business metadata to obtain the target business metadata; Through the above technical solution, when defining, business metadata objects and business metadata attributes are described from a perspective that can be easily understood by business personnel, and technical metadata that describes business metadata mapping is established, and the mapping relationship between the two is established. Then, through the mapping relationship between technical metadata and entity objects, business metadata and database entities are associated. Finally, after the business metadata defined according to the business is saved, the corresponding usage threshold can be lowered when users retrieve data. Similarly, the metadata definition instructions of the target user may be visual instructions, such as drag / click / input definition, etc.

[0007] Step S3, generating a business data set according to the graphical construction instruction from the target user, the target business metadata and the first target relational data source, wherein the business data set includes a first logic script representing the second target relational data source, the target business metadata object, the target association relationship and the target component object; Through the above technical solution, the user's behavior of graphically constructing a data set can be read and saved, and a first logic script that can be further compiled can be obtained as a business data set, so that in subsequent steps, when data needs to be obtained, the business data set can be directly referenced, and the first logic script can be further compiled to obtain relevant information; Similarly, the graphical construction instructions of the target user may be visual instructions, such as dragging / clicking / inputting definitions, etc.

[0008] Step S4: according to the graphical data acquisition instruction from the target user, the business data set is referenced to generate a second logic script for self-service data acquisition in the second target relational data source as a data acquisition task; Through the above technical solution, the user's graphical data acquisition behavior can be read and saved to obtain a second logic script that can be further compiled as a data acquisition task; the user does not need to use SQL coding, and can describe his own data acquisition needs through graphical data acquisition instructions, which reduces the difficulty of data acquisition; Similarly, the graphical data acquisition instructions of the target user may be visual instructions, such as dragging / clicking / input definition, etc.

[0009] Step S5: according to the target data source type of the second target relational data source and the target physical script syntax of the target data source type, parse the first logic script to generate a first physical script, and parse the second logic script to generate a second physical script; Through the above technical solution, by selecting the appropriate target physical script syntax according to the target data source type, the accuracy of the generated physical script can be improved to better perform the data acquisition task.

[0010] Step S6: assemble the first physics script and the second physics script to obtain a third physics script and submit the third physics script to the second target relational data source; Through the above technical solution, a complete data acquisition script can be obtained, and after submitting it to the corresponding relational data source, the physical query plan is further parsed to obtain the desired data.

[0011] Step S7, in response to the third physics script being executed successfully, downloading the target data corresponding to the data acquisition task; Through the above technical solution, after successful execution, the corresponding relational data source will return a successful result, and the user will be able to download the data he wants to use.

[0012] In order to better implement the present invention, further, step S3 includes: The target business metadata objects, target association relationships and target component objects selected by the graphical construction instruction in the target business metadata, as well as the second target relational data source selected in the first target relational data source, are saved in a preset business data set DSL configuration template to obtain the first logic script.

[0013] Through the above technical solution, a DSL configuration template is used to generate the first logic script. When making presets, more complex operations can be implemented with a small amount of code, thereby simplifying the development process and improving development efficiency. At the same time, the DSL configuration template has good adaptability and scalability. With the expansion of the first target relational data source, the DSL configuration template can be customized and expanded according to new requirements in specific fields, so that it can better cope with changes.

[0014] In order to better implement the present invention, further, step S3 includes: Step S301: According to the selected second target relational data source, obtain the corresponding first type and second type, wherein the second type is a subtype of the first type; Step S302: in response to the entity business metadata object and the relationship business metadata object being dragged into the preset directory tree, the selected entity business metadata object and the relationship business metadata object are saved as the target business metadata object; Step S303: in response to the first association attribute being dragged into the second association attribute, saving the association relationship between the primary business metadata object and the secondary business metadata object as the target association relationship, and at the same time, generating and saving the first technical attribute corresponding to the target association relationship according to the first mapping relationship and the target association relationship; Step S304: Acquire the business attributes of the business metadata object, and save the business attributes and the second technical attributes corresponding to the business attributes into an object list as the constituent objects.

[0015] Through the above technical solution, the relevant data configured by the graphical construction instructions of the user to construct the data set can be efficiently saved to the DSL configuration template through the DSL configuration template to generate the first logic script; The first type is the database types divided according to the major database categories, and the second type is the database types subdivided under the major categories; When adding a business metadata object to a business data set, the actual operation on the user side is to select the business metadata object based on the depicted business metadata object directory tree and drag it into the graphical canvas, and then the related objects will be saved according to the DSL configuration template; When creating an association relationship between business metadata objects for a business data set, the actual operation on the user side is to drag the association attribute of the main business metadata object and drag it to the association attribute of the secondary business metadata object, thereby establishing an association relationship between the two business metadata objects and describing the data range of the two association objects from a business perspective. When the association relationship based on the business metadata object is defined on the front end, the back end will also automatically generate and save the technical attributes of the association relationship in the DSL script according to the mapping relationship between the business metadata and the technical metadata; When setting the component objects for a business data set, the business attributes selected by the user as business metadata objects will be automatically added to the list of component objects of the data set. At the same time, when saving the business attributes, the technical attributes corresponding to the business attributes are automatically parsed and saved based on the mapping relationship between business metadata and technical metadata. They may include the corresponding field ID, field name, field data type, stored table ID, and stored table name.

[0016] In order to better implement the present invention, further, step S304 also includes: Automatically classify the technical attributes according to the field data type value of the technical attributes; If the field data type value of the technical attribute is a dictionary value attribute, the classification result of the technical attribute is converted into a dictionary value name.

[0017] Through the above technical solution, a more detailed DSL logic script can be obtained, and when the DSL logic script is subsequently compiled and adapted, a more accurate physical script can be obtained; when classifying, fields such as text and date will be automatically classified as attributes, and fields of numeric type will be automatically classified as values; At the same time, add "Is it a dictionary" and "Is it a dictionary conversion" configurations in the template. If it is a dictionary value attribute, by setting the automatic conversion dictionary name switch, the final attribute data acquisition result will be automatically generated as a sub-statement associated with the dictionary query and converted to a dictionary value name, further improving the accuracy of the physical script generated after the DSL logic script is compiled.

[0018] In order to better implement the present invention, further, the data acquisition task is configured into multiple subtasks.

[0019] The above technical solution can improve the flexibility and scalability of data retrieval, while improving user experience and supporting complex queries. It can more easily adapt to different user needs and data structures. When new functions need to be added or new data types need to be supported, only the corresponding subtasks need to be expanded or modified.

[0020] In order to better implement the present invention, further, step S4 includes: According to the graphical construction instruction, a target data column subtask, a target calculation formula subtask, a target filtering condition subtask and a target sorting method subtask are obtained; The second logic script is generated according to the preset DSL semantic data acquisition template, the target data column subtask, the target calculation formula subtask, the target filtering condition subtask and the target sorting method subtask.

[0021] Through the above technical solution, the target data column subtask is used to determine the specific data column that the user needs to extract from the database, the target calculation formula subtask is used to define how to calculate or convert the extracted data column to meet the specific needs of the user, the target filtering condition subtask is used to set the conditions for extracting data to filter out data rows that meet the user's requirements, and the target sorting method subtask is used to determine the sorting method of the extracted data so that the user can view and analyze the data more conveniently; The target filtering condition subtask can significantly reduce the amount of data that needs to be processed, thereby improving query efficiency and optimizing query performance. By combining different subtasks, the system can support complex query requirements. For example, users can set multiple filtering conditions, calculation formulas, and sorting methods at the same time to meet specific data analysis needs and support complex queries. The use of DSL semantic data acquisition template also improves the adaptability and scalability when configuring data acquisition tasks.

[0022] In order to better implement the present invention, further, step S5 includes: According to preset logic script grammar rules, checking whether the grammar of the first logic script and the second logic script is correct; If the syntax of the first logic script and the second logic script are both correct, acquiring the corresponding target physical script syntax according to the target data source type of the second target relational data source; Generate a first AST tree corresponding to the first logic script according to the first mapping relationship, the target association relationship and the first technical attribute; Compile the second logic script according to the business attribute corresponding to the data acquisition task and the second technical attribute corresponding to the business attribute to obtain a second AST tree; According to the labeled physics script syntax and the first AST tree, a first sub-statement representing the associated condition is generated as the first physics script; according to the labeled physics script syntax and the second AST tree, a second sub-statement representing the data acquisition task is generated as the second physics script.

[0023] Through the above technical solution, the AST tree is used as the intermediate representation layer of the logical script, providing a bridge from user input to physical script execution, and then being able to understand and process user requests in a structured manner; In the process of generating the AST tree, the grammatical and semantic integrity of the logical script can be checked, which helps to capture the potential AST tree before generating the physical script, providing a basis for subsequent query optimization and execution plan generation. The system can select the best query path and execution strategy errors based on the structure and properties of the AST tree, thereby improving the overall robustness; The AST tree provides a basis for subsequent query optimization and execution plan generation. The best query path and execution strategy can be selected based on the structure and properties of the AST tree. Therefore, through the AST tree, it is easier to support different database types. Only by adjusting the generation rules of the AST tree and the generation logic of the physical script, support for new database types can be achieved, further improving flexibility and scalability; the AST tree can represent complex query structures and nested relationships, which is conducive to processing complex user-defined data retrieval requirements; by optimizing the structure and properties of the AST tree, a more efficient physical script can be generated, thereby improving the execution efficiency of data retrieval.

[0024] In a second aspect, the present application provides a self-service data retrieval system based on business metadata semantic visualization, including: A data source management module, the data source management module is used to determine the first target relational data source and obtain the data source type of each data source and the physical script syntax corresponding to the data source type according to the data source definition instruction from the target user; A metadata management module, the metadata management module is used to save the business metadata defined according to the business direction and save the first mapping relationship between the technical metadata and the business metadata according to the metadata definition instruction from the target user, so as to obtain the target business metadata; A data set management module, the data set management module is used to generate a business data set according to a graphical construction instruction from a target user, the target business metadata and the first target relational data source, the business data set including a first logic script representing a second target relational data source, a target business metadata object, a target association relationship and a target component object; A data acquisition task management module, the data acquisition task management module is used to refer to the business data set according to the graphical data acquisition instruction from the target user, and generate a second logic script for self-service data acquisition in the second target relational data source as a data acquisition task; A first data acquisition task execution module, the first data acquisition task execution module is used to parse the first logic script to generate a first physical script according to the target data source type of the second target relational data source and the target physical script syntax of the target data source type, and parse the second logic script to generate a second physical script; A second data acquisition task execution module, the second data acquisition task execution module is used to assemble the first physics script and the second physics script, obtain a third physics script and submit the third physics script to the second target relational data source; A third data acquisition task execution module, wherein the third data acquisition task execution module is used to download target data corresponding to the data acquisition task in response to successful execution of the third physics script.

[0025] In order to better implement the present invention, further, it also includes: Visual development module, data acquisition task monitoring module, data acquisition template management module, calculation engine module, scheduling task management module and system management module; The visual development module is used to provide a graphical operation interface for target users; The data acquisition task monitoring module is used to monitor the data acquisition results of the data acquisition task; The data acquisition template management module is used to edit the default template of the data acquisition task; The computing engine module is used to store physical script syntax and exchange data with a data source; The scheduling task management module is used to configure the automatic data acquisition task; The system management module is used to configure the system's permission control, operation audit and security management and control.

[0026] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) Reduce the user's technical threshold cost and daily operation and maintenance costs, so that business personnel can also retrieve data from the relational database by themselves according to their own needs; (2) It shields the differences between the underlying relational databases and improves the flexibility, security, and accuracy of self-service data retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention is further described in conjunction with the following drawings and embodiments, and all the concepts and innovations of the present invention should be regarded as the disclosed contents and the protection scope of the present invention.

[0028] Figure 1 The present invention is a flowchart of an embodiment of a method for visually generating self-service data retrieval based on business metadata semantics in the present application.

[0029] Figure 2 This is a flowchart of another embodiment of a method for visually generating self-service data retrieval based on business metadata semantics in the present application.

[0030] Figure 3 This is a structural diagram of an embodiment of a self-service data retrieval system based on business metadata semantic visualization in this application.

[0031] Figure 4 This is a structural diagram of another embodiment of a self-service data retrieval system based on business metadata semantic visualization in this application. DETAILED DESCRIPTION

[0032] Embodiment 1: With the continuous growth of enterprise data and the increasing complexity of business needs, the existing data acquisition methods based on technical metadata and direct connection to the database through SQL coding can no longer meet the needs of business personnel for fast and flexible data acquisition. Therefore, a self-service data acquisition method based on business metadata semantic visualization has emerged, aiming to help business personnel easily achieve self-service data acquisition through semantic understanding and visualization interface.

[0033] A self-service data retrieval method based on business metadata semantic visualization is provided in this embodiment. Figure 1 As shown, the following steps are included: Step S1: according to the data source definition instruction from the target user, determine the first target relational data source and obtain the data source type of each data source and the physical script syntax corresponding to the data source type; Specifically, in this embodiment, the data source type includes a first type and a second type, the first type includes MySQL category, PostgreSQL category and other categories, and the second type includes specific data source types under each first type, such as TeleDB under MySQL category, TelePG under PostgreSQL category, and DAMO under other categories; For the physical script syntax of the MySQL category, the syntax will be different for character sets and database operations. For example, MySQL allows you to specify character sets when creating a database, such as GBK. For the physical script syntax of the PostgreSQL category, the syntax will be different for table operations and data types. For example, in the PostgreSQL category, the ALTER TABLE statement is used to rename a table.

[0034] Step S2: according to the metadata definition instruction from the target user, save the business metadata defined according to the business direction and save the first mapping relationship between the technical metadata and the business metadata to obtain the target business metadata; Specifically, business metadata defined in terms of business can be defined according to actual needs, such as "resources", "hardware", "links" and other definition naming methods that are easy for business personnel to understand. After completing the definition and the mapping with the technical metadata, the obtained metadata is the target business metadata; since there is a second mapping relationship between the technical metadata and the entity, after completing the definition, the entity can be adjusted by adjusting the target business metadata during subsequent graphical construction and graphical data acquisition.

[0035] Step S3, generating a business data set according to the graphical construction instruction from the target user, the target business metadata and the first target relational data source, wherein the business data set includes a first logic script representing the second target relational data source, the target business metadata object, the target association relationship and the target component object; Specifically, the second target relational data source is a specific relational data source that needs to perform the data acquisition task; the target business metadata object may include relevant parameters such as ID, name, coordinates, etc.; the target association relationship is the association relationship required for data acquisition, which may include direct association relationships between different business objects, or association relationships between business objects and technical objects; the target component objects may include attribute objects and value objects of metadata, etc.

[0036] Step S4: according to the graphical data acquisition instruction from the target user, the business data set is referenced to generate a second logic script for self-service data acquisition in the second target relational data source as a data acquisition task; Optionally, in addition to the graphical data acquisition instructions directly input by the target user, a semantic model can also be set to understand the text data acquisition requirements input by the user, and then generate a second logic script for self-service data acquisition based on the understood data acquisition requirements; This semantic model can be a rule-based model or a machine learning model, such as a natural language processing model based on deep learning.

[0037] Step S5: according to the target data source type of the second target relational data source and the target physical script syntax of the target data source type, parse the first logic script to generate a first physical script, and parse the second logic script to generate a second physical script; Specifically, when the second target relational data source is TeleDB, its first type is the MySQL category and its second type is TeleDB. If the preset physical script syntax contains both the MySQL category syntax and the TeleDB syntax, the physical script syntax corresponding to TeleDB is preferentially selected to generate the first physical script and the second physical script. When parsing, preset grammar rules and lexical rules may be used to write and generate a parser, the parser may be used to verify the grammar of the logic script, and the parser may be used to further parse the logic script to generate a physical script.

[0038] Step S6: assemble the first physics script and the second physics script to obtain a third physics script and submit the third physics script to the second target relational data source; Step S7, in response to the third physics script being executed successfully, downloading the target data corresponding to the data acquisition task; Specifically, after submitting the third physical script to the second target relational data source, the second target relational data source will further parse the third physical script to obtain a physical query plan. When the script is correct, a successful receipt will be returned and the template user can download the required data.

[0039] Embodiment 2: This embodiment is further optimized on the basis of Embodiment 1. In this embodiment, step S3 includes: The target business metadata objects, target association relationships and target component objects selected by the graphical construction instruction in the target business metadata, as well as the second target relational data source selected in the first target relational data source, are saved in a preset business data set DSL configuration template to obtain the first logic script.

[0040] In order to better implement the present invention, further, refer to Figure 2 , the step S3 comprises: Step S301: According to the selected second target relational data source, obtain the corresponding first type and second type, wherein the second type is a subtype of the first type; Step S302: in response to the entity business metadata object and the relationship business metadata object being dragged into the preset directory tree, the selected entity business metadata object and the relationship business metadata object are saved as the target business metadata object; Step S303: in response to the first association attribute being dragged into the second association attribute, saving the association relationship between the primary business metadata object and the secondary business metadata object as the target association relationship, and at the same time, generating and saving the first technical attribute corresponding to the target association relationship according to the first mapping relationship and the target association relationship; Step S304: Acquire the business attributes of the business metadata object, and save the business attributes and the second technical attributes corresponding to the business attributes into an object list as the constituent objects.

[0041] Furthermore, the step S304 further includes: Automatically classify the technical attributes according to the field data type value of the technical attributes; If the field data type value of the technical attribute is a dictionary value attribute, the classification result of the technical attribute is converted into a dictionary value name.

[0042] Specifically, in this embodiment, a DSL configuration template is as follows: Dataset: Data Source: Data source ID: <data source ID> Data source type: <Data source type> Business object: - <Unique ID>: Name: <Name> Coordinates: <X coordinate, Y coordinate> Width: <Width> Height: <Height> Association relationship: - Association type: <Left association | Right association | Inner association> Business association attribute: Main business object ID: <Main business object ID> Subordinate business object ID: <Subordinate business object ID> Association condition: <Complete business attribute association condition> Technical association attribute: Main physical object ID: <Main physical object ID> Subordinate physical object ID: <Subordinate physical object ID> Association condition: <Complete physical field association condition> Attribute object: - <Name>: Business attribute: Alias: <Alias> Business attribute ID: <Business attribute ID> Business object ID: <Belonging business object ID> Is it a dictionary: <Yes | No> Is dictionary conversion: <Yes | No> Technical attribute: Field ID: <Field ID> Field name: <Field name> Field data type: <Text | Numeric | Date> Table ID: Table name: Numeric object: - <Name>: Business attribute: Alias: <Alias> Business attribute ID: <Business attribute ID> Business object ID: <Belonging business object ID> Is it a dictionary: <Yes | No> Is dictionary conversion: <Yes | No> Technical attribute: Field ID: <Field ID> Field name: <Field name> Field data type: <text | value | date> Table ID: Table name: .

[0043] In this embodiment, step S4 includes: According to the graphical construction instruction, a target data column subtask, a target calculation formula subtask, a target filtering condition subtask and a target sorting method subtask are obtained; The second logic script is generated according to the preset DSL semantic data acquisition template, the target data column subtask, the target calculation formula subtask, the target filtering condition subtask and the target sorting method subtask.

[0044] Specifically, in this embodiment, a DSL semantic data acquisition template is as follows: Definition of number: - Dataset: Dataset unique ID: "" - Data columns: Component object name: Unique ID of the component object: "" Aliases: "" order: "" Data Type: "" Display format: Format: "" Precision: "" Unit: "" Calculation formula: "" - Sorting: Component object name: Unique ID of the component object: "" sort by: "" Priority: "" - Filter: Component object name: Unique ID of the component object: "" Filter by: "".

[0045] In this embodiment, step S5 includes: According to preset logic script grammar rules, checking whether the grammar of the first logic script and the second logic script is correct; If the syntax of the first logic script and the second logic script are both correct, acquiring the corresponding target physical script syntax according to the target data source type of the second target relational data source; Generate a first AST tree corresponding to the first logic script according to the first mapping relationship, the target association relationship and the first technical attribute; Compile the second logic script according to the business attribute corresponding to the data acquisition task and the second technical attribute corresponding to the business attribute to obtain a second AST tree; According to the labeled physics script syntax and the first AST tree, a first sub-statement representing the associated condition is generated as the first physics script; according to the labeled physics script syntax and the second AST tree, a second sub-statement representing the data acquisition task is generated as the second physics script.

[0046] Specifically, this embodiment verifies the accuracy of the first logic script and the second logic script by setting a DSL parser, and generates a first AST tree and a second AST tree; When parsing the first logic script and the second logic script, the business object relationship can be obtained by parsing the first logic script, and after further syntax adaptation, an associated conditional statement can be generated; When parsing the second logic script, the target data column subtask, target calculation formula subtask, target filtering condition subtask and target sorting method subtask are parsed to obtain the corresponding physical properties. For the target calculation formula subtask, target filtering condition subtask and target sorting method subtask, syntax adaptation is performed directly according to the obtained physical properties to generate a grouping statement corresponding to the target calculation formula subtask, a conditional filtering statement corresponding to the target filtering condition subtask, and a sorting statement corresponding to the target sorting method subtask; For the target data column subtask, after obtaining its physical attributes, it is also necessary to determine whether to convert it into a dictionary name. If conversion is required, it is associated with the dictionary table to generate a subquery field, and after syntax adaptation, a complete query field statement is generated. If conversion is not required, a physical field is generated, and syntax adaptation is performed, and then a complete query field statement is generated. A second AST tree in JSON format in this embodiment is as follows: { "type": "Data definition", "children": [ { "type": "Dataset", "children": [ { "type": "Dataset unique ID", "value": "" } ] }, { "type": "Data Column", "children": [ { "type": "Name of component object", "children": [ { "type": "Unique ID of the component object", "value": "" }, { "type": "Alias", "value": "" }, { "type": "Sequential", "value": "" }, { "type": "Data type", "value": "" }, { "type": "Display format", "children": [ { "type": "Format", "value": "" }, { "type": "precision", "value": "" }, { "type": "Unit", "value": "" } ] }, { "type": "Calculation formula", "value": "" } ] } ] }, { "type": "Sort", "children": [ { "type": "Name of component object", "children": [ { "type": "Unique ID of the component object", "value": "" }, { "type": "Sort method", "value": "" }, { "type": "Priority", "value": "" } ] } ] }, { "type": "Filter", "children": [ { "type": "Name of component object", "children": [ { "type": "Unique ID of the component object", "value": "" }, { "type": "Filter Condition", "value": "" } ] } ] } ] }.

[0047] Specifically, in this embodiment, in step S6, the associated conditional statements, query field statements, grouping statements, conditional filtering statements and sorting statements obtained in step S5 will be assembled to obtain a third physical script, and the third physical script will be subjected to another syntax check before being submitted to the second target relational data source.

[0048] The other parts of this embodiment are the same as those of Embodiment 1, and thus will not be described in detail.

[0049] In summary, the beneficial effects of this embodiment also include: The DSL configuration template and DSL semantic data retrieval template simplify the development process and improve development efficiency. They also have good adaptability and scalability to better cope with different types of relational data sources. Further consideration is given to the association between business metadata objects, field data type values ​​of technical attributes, etc., which further improves the accuracy of the physical script generated after the DSL logic script is compiled; Using the AST tree as the intermediate representation layer of the logic script further improves flexibility and scalability, facilitates processing of complex data retrieval requirements defined by users, and improves the execution efficiency of data retrieval.

[0050] Embodiment 3: This embodiment provides a self-service data acquisition system based on business metadata semantic visualization generation. Figure 3 ,include: A data source management module, the data source management module is used to determine the first target relational data source and obtain the data source type of each data source and the physical script syntax corresponding to the data source type according to the data source definition instruction from the target user; A metadata management module, the metadata management module is used to save the business metadata defined according to the business direction and save the first mapping relationship between the technical metadata and the business metadata according to the metadata definition instruction from the target user, so as to obtain the target business metadata; A data set management module, the data set management module is used to generate a business data set according to a graphical construction instruction from a target user, the target business metadata and the first target relational data source, the business data set including a first logic script representing a second target relational data source, a target business metadata object, a target association relationship and a target component object; A data acquisition task management module, the data acquisition task management module is used to refer to the business data set according to the graphical data acquisition instruction from the target user, and generate a second logic script for self-service data acquisition in the second target relational data source as a data acquisition task; A first data acquisition task execution module, the first data acquisition task execution module is used to parse the first logic script to generate a first physical script according to the target data source type of the second target relational data source and the target physical script syntax of the target data source type, and parse the second logic script to generate a second physical script; A second data acquisition task execution module, the second data acquisition task execution module is used to assemble the first physics script and the second physics script, obtain a third physics script and submit the third physics script to the second target relational data source; A third data acquisition task execution module, wherein the third data acquisition task execution module is used to download target data corresponding to the data acquisition task in response to successful execution of the third physics script.

[0051] Embodiment 4: This embodiment is further optimized on the basis of the above embodiment 3. Figure 4 , a self-service data acquisition system based on business metadata semantic visualization, also includes: Visual development module, data acquisition task monitoring module, data acquisition template management module, calculation engine module, scheduling task management module and system management module; The visual development module is used to provide a graphical operation interface for target users; The data acquisition task monitoring module is used to monitor the data acquisition results of the data acquisition task; The data acquisition template management module is used to edit the default template of the data acquisition task; The computing engine module is used to store physical script syntax and exchange data with a data source; The scheduling task management module is used to configure the automatic data acquisition task; The system management module is used to configure the system's permission control, operation audit and security management and control.

[0052] Specifically, in this embodiment, the first data acquisition task execution module, the second data acquisition task execution module and the third data acquisition task execution module are integrated together to form a data acquisition task execution module; It also includes a homepage module, which together constitutes the application layer of the system: a homepage module, a data source management module, a metadata management module, a data set management module, a data acquisition task management module, a data acquisition task execution module, a visualization development module, a data acquisition task monitoring module, and a data acquisition template management module; The system management module includes the permission control submodule, the operation audit submodule and the security control submodule, and like the scheduling task management module, they are all set up separately; The computing engine module is provided with a MySQL engine submodule, a PostgreSQL engine submodule, other engine submodules and an automatic sensing adaptation layer submodule, each engine submodule stores a corresponding physical script syntax, and the automatic sensing adaptation layer submodule is used to identify the type of docked data source; The other parts of this embodiment are the same as those of the above-mentioned embodiment 3, so they will not be described in detail.

[0053] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A self-service data acquisition method based on business metadata semantic visualization, characterized in that: The following steps are involved: Step S1: according to the data source definition instruction from the target user, determine the first target relational data source and obtain the data source type of each data source and the physical script syntax corresponding to the data source type; Step S2: according to the metadata definition instruction from the target user, save the business metadata defined according to the business direction and save the first mapping relationship between the technical metadata and the business metadata to obtain the target business metadata; Step S3, generating a business data set according to the graphical construction instruction from the target user, the target business metadata and the first target relational data source, wherein the business data set includes a first logic script representing the second target relational data source, the target business metadata object, the target association relationship and the target component object; Step S4: according to the graphical data acquisition instruction from the target user, the business data set is referenced to generate a second logic script for self-service data acquisition in the second target relational data source as a data acquisition task; Step S5: according to the target data source type of the second target relational data source and the target physical script syntax of the target data source type, parse the first logic script to generate a first physical script, and parse the second logic script to generate a second physical script; Step S6: assemble the first physics script and the second physics script to obtain a third physics script and submit the third physics script to the second target relational data source; Step S7: In response to the successful execution of the third physical script, download the target data corresponding to the data acquisition task.

2. According to claim 1, a self-service data acquisition method based on business metadata semantic visualization generation is characterized in that: The step S3 comprises: The target business metadata objects, target association relationships and target component objects selected by the graphical construction instruction in the target business metadata, as well as the second target relational data source selected in the first target relational data source, are saved to a preset business data set DSL configuration template to obtain the first logic script.

3. The method for self-service data retrieval based on business metadata semantic visualization according to claim 2 is characterized in that: The step S3 comprises: Step S301: According to the selected second target relational data source, obtain the corresponding first type and second type, wherein the second type is a subtype of the first type; Step S302: in response to the entity business metadata object and the relationship business metadata object being dragged into the preset directory tree, saving the selected entity business metadata object and the relationship business metadata object as the target business metadata object; Step S303: in response to the first association attribute being dragged into the second association attribute, saving the association relationship between the primary business metadata object and the secondary business metadata object as the target association relationship, and at the same time, generating and saving the first technical attribute corresponding to the target association relationship according to the first mapping relationship and the target association relationship; Step S304: Acquire the business attributes of the business metadata object, and save the business attributes and the second technical attributes corresponding to the business attributes into an object list as the constituent objects.

4. The method for self-service data retrieval based on business metadata semantic visualization according to claim 3 is characterized in that: The step S304 further includes: Automatically classify the technical attributes according to the field data type value of the technical attributes; If the field data type value of the technical attribute is a dictionary value attribute, the classification result of the technical attribute is converted into a dictionary value name.

5. According to claim 1, a self-service data acquisition method based on business metadata semantic visualization generation is characterized in that: The data acquisition task is configured into multiple subtasks.

6. The method for self-service data retrieval based on business metadata semantic visualization according to claim 5, characterized in that: The step S4 comprises: According to the graphical construction instruction, a target data column subtask, a target calculation formula subtask, a target filtering condition subtask and a target sorting method subtask are obtained; The second logic script is generated according to the preset DSL semantic data acquisition template, the target data column subtask, the target calculation formula subtask, the target filtering condition subtask and the target sorting method subtask.

7. The method for self-service data retrieval based on business metadata semantic visualization according to claim 3 is characterized in that: The step S5 comprises: According to preset logic script grammar rules, checking whether the grammar of the first logic script and the second logic script is correct; If the syntax of the first logic script and the second logic script are both correct, acquiring the corresponding target physical script syntax according to the target data source type of the second target relational data source; Generate a first AST tree corresponding to the first logic script according to the first mapping relationship, the target association relationship and the first technical attribute; Compile the second logic script according to the business attribute corresponding to the data acquisition task and the second technical attribute corresponding to the business attribute to obtain a second AST tree; According to the labeled physics script syntax and the first AST tree, a first sub-statement representing the associated condition is generated as the first physics script; according to the labeled physics script syntax and the second AST tree, a second sub-statement representing the data acquisition task is generated as the second physics script.

8. A self-service data retrieval system based on business metadata semantic visualization, characterized in that: include: A data source management module, the data source management module is used to determine the first target relational data source and obtain the data source type of each data source and the physical script syntax corresponding to the data source type according to the data source definition instruction from the target user; A metadata management module, the metadata management module is used to save the business metadata defined according to the business direction and save the first mapping relationship between the technical metadata and the business metadata according to the metadata definition instruction from the target user, so as to obtain the target business metadata; A data set management module, the data set management module is used to generate a business data set according to a graphical construction instruction from a target user, the target business metadata and the first target relational data source, the business data set including a first logic script representing a second target relational data source, a target business metadata object, a target association relationship and a target component object; A data acquisition task management module, the data acquisition task management module is used to refer to the business data set according to the graphical data acquisition instruction from the target user, and generate a second logic script for self-service data acquisition in the second target relational data source as a data acquisition task; A first data acquisition task execution module, the first data acquisition task execution module is used to parse the first logic script to generate a first physical script according to the target data source type of the second target relational data source and the target physical script syntax of the target data source type, and parse the second logic script to generate a second physical script; A second data acquisition task execution module, the second data acquisition task execution module is used to assemble the first physics script and the second physics script, obtain a third physics script and submit the third physics script to the second target relational data source; A third data acquisition task execution module, wherein the third data acquisition task execution module is used to download target data corresponding to the data acquisition task in response to successful execution of the third physics script.

9. The self-service data acquisition system based on business metadata semantic visualization generation according to claim 8 is characterized in that: Also includes: Visual development module, data acquisition task monitoring module, data acquisition template management module, calculation engine module, scheduling task management module and system management module; The visual development module is used to provide a graphical operation interface for target users; The data acquisition task monitoring module is used to monitor the data acquisition results of the data acquisition task; The data acquisition template management module is used to edit the default template of the data acquisition task; The computing engine module is used to store physical script syntax and exchange data with a data source; The scheduling task management module is used to configure the automatic data acquisition task; The system management module is used to configure the system's permission control, operation audit and security management and control.

Images