New field access method and system for data set and electronic equipment

Abstract

The invention relates to the technical field of data sets, and discloses a new field access method and system for a data set and electronic equipment. According to the application, a server takes an original field and an existing target field as superior fields, generates a new target field according to a field generation rule, and generates a field chain table according to the field generation rule between the target fields, so that access statement fragments corresponding to the target fields are combined in a WITH clause expression mode according to the field chain table, and access statement fragments corresponding to the target fields are obtained. According to the method, the field access statement is obtained, the access service of the target fields is provided for the user terminal through the field access statement, organization and arrangement between the target fields are achieved through the field linked list, the number of the database objects is controlled to be one, the data architecture is greatly simplified, field expansion can be achieved without stopping a data set, and the user experience is improved. Therefore, the query statement is more concise and efficient, the data set expansion steps are reduced from two aspects, and the data set expansion efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data set technology, and in particular to a method, system and electronic device for accessing a newly added field of a data set. Background Art

[0002] In the current wave of digital transformation, the rapid development of information technology and the widespread application of big data have greatly promoted the improvement of corporate decision-making efficiency and business innovation capabilities. As a core production factor, the efficient management and flexible application of data have become the key to corporate competitiveness. However, with the increase in business complexity and the increasing diversification of data analysis needs, traditional data set management methods have gradually exposed their limitations.

[0003] Traditional data set architecture is often based on static design, that is, the field structure and data model of the data set are fixed at the beginning of creation, which makes it difficult to adapt to subsequent business changes and the rapid growth of data analysis needs. When the project is upgraded or the business needs change, if the data set fields need to be expanded, it is usually necessary to go through a cumbersome process: first, the use of the data set needs to be suspended to avoid data inconsistency; then, the developer needs to go deep into the bottom layer, modify the SQL query statement of the data set, and even upload a new data source file. This process is not only time-consuming and labor-intensive, but may also cause service interruption and affect business continuity. In order to alleviate this problem, some companies try to build a complex data warehouse system and use multiple database tables or views for hierarchical storage and logical processing of data to achieve dynamic expansion of data sets. However, although this method improves the flexibility of the data set to a certain extent, the newly added database tables or views not only increase the complexity and maintenance cost of the system, but also require fine management of data permissions to ensure data security and compliance, making the expansion process of the data set cumbersome and error-prone, and the utilization efficiency and response speed of the data set are still low.

[0004] Therefore, the existing dataset expansion methods have the disadvantage of cumbersome steps due to various reasons, resulting in low dataset expansion efficiency, which in turn affects work efficiency. Summary of the invention

[0005] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0006] In view of the above-mentioned shortcomings of the prior art, the present application discloses a method, system and electronic device for accessing newly added fields of a data set, so as to improve the expansion efficiency of the data set.

[0007] The present application discloses a method for accessing newly added fields of a data set, which is applied to a server side, and the server side is connected to a user terminal. The method comprises: obtaining an original data set, wherein the original data set comprises original fields; in response to a field generation rule corresponding to the original data set, generating a new target field according to the original field and / or an existing target field according to the field generation rule; establishing a linked list data structure according to the field generation rule between each target field, and obtaining a field linked list composed of each target field, wherein the field linked list stores access statement fragments corresponding to the target field; according to the field linked list, combining the access statement fragments in a WITH clause expression to obtain a field access statement, wherein the user terminal is used to access the target field through the field access statement.

[0008] In one embodiment of the present application, a new target field is generated according to the original field and / or the existing target field according to the field generation rule, including at least one of the following: if the rule type corresponding to the field generation rule is a calculated column type, the field generation rule includes a field calculation statement and a field type, the field calculation statement is executed according to the original field and / or the existing target field to obtain a calculation result, and the calculation result is verified according to the field generation rule. After the verification passes, the calculation result is stored according to the field type to obtain the target field, wherein the field calculation statement is generated based on an SQL statement; if the rule type corresponding to the field generation rule is an aggregate column type, the field generation rule includes a window function, the window function is executed according to the original field and / or the existing target field to obtain the target field output by the window function.

[0009] In one embodiment of the present application, a linked list data structure is established according to the field generation rules between each target field to obtain a field linked list composed of each target field, including: obtaining one or more business items; classifying each target field according to the business item to obtain global fields and private fields, wherein the private field includes project fields corresponding to each business item respectively; establishing a linked list data structure according to the field generation rules between each global field to obtain a global field chain, and establishing a linked list data structure according to the field generation rules between each project field to obtain a project field chain corresponding to each business item respectively; linking the global field chain to each project field chain according to the field generation rules to obtain a complete field chain.

[0010] In one embodiment of the present application, each of the target fields is classified according to the business project to obtain global fields and private fields, including: pre-setting usage probability information corresponding to each of the original fields, wherein the usage probability information includes the field usage probability corresponding to each of the business projects; calculating according to the usage probability information corresponding to the first field to obtain the usage probability information corresponding to the second field, wherein the first field is the parent field corresponding to the second field, and the second field is any target field; arranging the field usage probabilities corresponding to the second field from large to small, and determining the first priority probability and the second priority probability; if the difference between the first priority probability and the second priority probability is greater than or equal to a preset threshold, determining the second field as the project field corresponding to the pending project, wherein the pending project is the business project corresponding to the first priority probability; if the difference between the first priority probability and the second priority probability is less than the preset threshold, determining the second field as a global field.

[0011] In one embodiment of the present application, according to the field chain list, the access statement fragments are combined in a WITH clause expression to obtain a field access statement, including: obtaining a target project, wherein the target project is any business project; determining the current field chain corresponding to the target project from each of the project field chains according to the matching result between the target project and the business project, and combining the global field chain and the current field chain to obtain a field chain to be accessed; according to the link relationship between each target field in the field chain to be accessed, the access statement fragments in the field chain to be accessed are combined in sequence in a WITH clause expression, and a query main sentence is generated according to the access statement fragments to obtain a field access statement composed of the access statement fragments and the query main sentence.

[0012] In one embodiment of the present application, the access statement fragments in the field chain to be accessed are combined in sequence using a WITH clause expression, including: if the target field does not belong to an aggregate field, then the access statement fragments corresponding to the target field are combined; if the target field belongs to an aggregate field, then when generating a query main sentence, the access statement fragments corresponding to the aggregate field are combined in the query main sentence, wherein the field generation rule corresponding to the aggregate field includes an aggregate function.

[0013] In one embodiment of the present application, obtaining an original data set includes: responding to a data source connection rule, wherein the data source connection rule includes a data source type and / or data source connection information, and connecting an original data source according to the data source connection information; responding to a data set generation rule, wherein the data set generation rule includes data source matching information, data collection information, and field storage information, matching from each of the original data sources according to the data source matching information to obtain a target data source, wherein the data source matching information includes a data set type and / or a data source identifier; collecting data from the target data source according to the data collection information to obtain a target data column; storing the target data column according to the field storage information to obtain an original field, and generating an original data set corresponding to the target data source according to the original field, wherein the field storage information includes a field type and / or a field name.

[0014] In one embodiment of the present application, the method also includes at least one of the following: receiving a data source connection rule sent by the user terminal, wherein the user terminal is used to display a data source interface so that the user can input the data source connection rule through the data source interface; receiving a data set generation rule sent by the user terminal, wherein the user terminal is used to display a data set interface so that the user can input the data set generation rule through the data set interface; receiving a field generation rule sent by the user terminal, wherein the user terminal is used to display a field interface so that the user can input the field generation rule through the field interface.

[0015] The present application discloses a newly added field access system for a data set, comprising: a server side, for acquiring an original data set, wherein the original data set includes original fields; in response to a field generation rule corresponding to the original data set, generating a new target field according to the original field and / or existing target field according to the field generation rule; establishing a linked list data structure according to the field generation rule between each target field, and obtaining a field linked list composed of each target field, wherein the field linked list stores access statement fragments corresponding to the target field; according to the field linked list, combining the access statement fragments in a WITH clause expression to obtain a field access statement; a user terminal, connected to the server side, and the user terminal is used to access the target field through the field access statement.

[0016] The present application discloses an electronic device, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above method.

[0017] Beneficial effects of this application:

[0018] The server uses the original field and the existing target field as the parent field, generates a new target field according to the field generation rules, and generates a field chain list according to the field generation rules between the target fields, so as to combine the access statement fragments corresponding to the target field in the WITH clause expression according to the field chain list to obtain the field access statement, and provide the target field access service to the user terminal through the field access statement. In this way, not only the organization and arrangement between the target fields is realized through the field chain list, but also the number of database objects is controlled to one compared to creating multiple database tables or views to realize field expansion, which greatly simplifies the data architecture. Moreover, the field access statement corresponding to the target field is generated according to the field reference relationship through the WITH clause, and the field expansion can be realized without deactivating the data set, making the query statement more concise and efficient, thereby reducing the data set expansion steps from two aspects and improving the data set expansion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a structural diagram of an application environment for implementing a newly added field access method for a data set in an embodiment of the present application;

[0020] Figure 2 It is a flowchart of a method for accessing a newly added field of a data set in an embodiment of the present application;

[0021] Figure 3 It is a flowchart of a method for obtaining data source connection rules in an embodiment of the present application;

[0022] Figure 4 It is a structural diagram of a data source interface in an embodiment of the present application;

[0023] Figure 5 It is a flowchart of a method for obtaining data set generation rules in an embodiment of the present application;

[0024] Figure 6 It is a structural diagram of a data set interface in an embodiment of the present application;

[0025] Figure 7 It is a flowchart of a method for obtaining field generation rules in an embodiment of the present application;

[0026] Figure 8 It is a structural diagram of a field interface in an embodiment of the present application;

[0027] Fig. 9 It is a structural diagram of another field interface in an embodiment of the present application;

[0028] Fig.10 It is a schematic diagram of the structure of a complete field chain in an embodiment of the present application;

[0029] Fig.11 It is a structural diagram of a newly added field access system for a data set in an embodiment of the present invention;

[0030] Fig.12 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and sub-samples in the embodiments can be combined with each other without conflict.

[0032] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0033] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0034] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0035] Unless otherwise stated, the term "plurality" means two or more.

[0036] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0037] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0038] Combination Figure 1 As shown, an embodiment of the present disclosure provides an application environment for implementing a method for accessing a newly added field of a data set, including a server and a user terminal, wherein the server is connected to the client terminal via a network.

[0039] The server side is used for at least part of the following: obtaining an original data set, the original data set including original fields; generating new target fields according to the field generation rules corresponding to the original data set based on the original fields and / or existing target fields according to the field generation rules; establishing a linked list data structure according to the field generation rules between the target fields to obtain a field linked list composed of the target fields, wherein the field linked list stores access statement fragments corresponding to the target fields; according to the field linked list, combining the access statement fragments in a WITH clause expression to obtain a field access statement.

[0040] The user terminal is used to access the target field through the field access statement.

[0041] Combination Figure 2 As shown, an embodiment of the present disclosure provides a method for accessing a newly added field of a data set, which is applied to a server side, and the server side is connected to a user terminal. The method includes:

[0042] Step S201, obtaining an original data set;

[0043] The original data set includes original fields;

[0044] Step S202, in response to the field generation rule corresponding to the original data set, generating a new target field according to the original field and / or the existing target field according to the field generation rule;

[0045] Step S203, establishing a linked list data structure according to the field generation rule between the target fields, and obtaining a field linked list composed of the target fields;

[0046] The field linked list stores the access statement fragment corresponding to the target field, the field name corresponding to the target field, the field classification, the actual field type, the field type specified by the user, whether it contains an aggregate function, the preceding field, and the succeeding field;

[0047] The access statement fragment corresponding to the target field is an expression of a SQL (Structured Query Language) statement;

[0048] Step S204, according to the field linked list, the access statement fragments are combined in a WITH clause expression to obtain a field access statement;

[0049] The WITH clause is a way to define temporary result sets in SQL queries. It allows developers to create in-memory result sets that can be used multiple times in subsequent queries, thereby simplifying complex query logic and improving code readability and maintainability.

[0050] The user terminal is used to access the target field through a field access statement.

[0051] The newly added field access method for a data set provided by the embodiment of the present disclosure is adopted. The server uses the original field and the existing target field as the parent field, generates a new target field according to the field generation rule, and generates a field linked list according to the field generation rule between the target fields, so as to combine the access statement fragments corresponding to the target field in the WITH clause expression according to the field linked list to obtain the field access statement, and provide the target field access service to the user terminal through the field access statement. In this way, not only the organization and arrangement between the target fields is realized through the field linked list, and the number of database objects is controlled to one compared to creating multiple database tables or views to realize field expansion, which greatly simplifies the data architecture. Moreover, the field access statement corresponding to the target field is generated according to the field reference relationship through the WITH clause, and the field expansion can be realized without deactivating the data set, making the query statement more concise and efficient, thereby reducing the data set expansion steps from two aspects and improving the data set expansion efficiency.

[0052] In some embodiments, the embodiments of the present disclosure provide a method for accessing newly added fields of a data set, including: obtaining an original data set, wherein the original data set includes original fields; in response to a field generation rule corresponding to the original data set, determining an upper-level field from a basic field, and generating a target field according to the field generation rule based on the upper-level field, and adding the target field to the basic field, wherein the basic field also includes the original field; generating a field reference relationship between target fields according to the field generation rule, so as to establish a linked list data structure according to the field reference relationship, and obtain a field linked list composed of each target field; generating a field access statement containing a WITH clause according to the field reference relationship, so that the user terminal accesses the target field through the field access statement.

[0053] Optionally, obtaining the original data set includes: responding to a data source connection rule, the data source connection rule includes a data source type and / or data source connection information, and connecting the original data source according to the data source connection information; responding to a data set generation rule, the data set generation rule includes data source matching information, data collection information and field storage information, matching from each original data source according to the data source matching information to obtain a target data source, wherein the data source matching information includes a data set type and / or a data source identifier; collecting data from the target data source according to the data collection information to obtain a target data column; storing the target data column according to the field storage information to obtain the original field, and generating an original data set corresponding to the target data source according to the original field, wherein the field storage information includes a field type and / or a field name.

[0054] In some embodiments, the original data source stores all the information for establishing a database connection to provide a device or original media for certain required data, just as a file can be found in a file system by specifying a file name. By providing the correct data source name, the corresponding database connection can be queried, which is often used in data analysis, data integration, business intelligence (BI) and other data-driven applications.

[0055] In some embodiments, the original data sources include relational databases (RDBMS), NoSQL (Not Only SQL) databases, data warehouses, file storage, file storage, application programming interfaces (API), sensors and Internet of Things (IoT) devices, spreadsheets, etc.

[0056] In some embodiments, relational databases include MySQL, PostgreSQL, Oracle, and SQL Server, etc., wherein relational databases use tables to store structured data and support SQL for data query and operation; NoSQL databases include MongoDB, Cassandra, and Couchbase, etc., wherein NoSQL databases are used to store unstructured or semi-structured data and support high scalability and flexible data models; data warehouses include Amazon Redshift, Google BigQuery, and Snowflake, etc., wherein data warehouses are used to store and analyze large amounts of historical data, optimize query performance, and support complex data analysis and report generation; file storage includes files in formats such as CSV (Comma-Separated Values), JSON (JavaScript Object Notation), XML (Extensible Markup Language), and Parquet, wherein the files are stored in a local file system, cloud storage (such as Amazon S3, Google Cloud Storage), or a distributed file system (such as Hadoop HDFS); real-time data streams include Apache Kafka, Apache Flink, and Apache Spark Streaming, in which real-time data streams are used to process and analyze data streams generated in real time, and are suitable for application scenarios that require real-time decision-making and response; application programming interfaces are used to provide programmatic access to data from external systems or services; sensors and IoT devices are hardware devices used to generate and transmit real-time data, such as temperature sensors, smart home devices, etc.; spreadsheets include Microsoft Excel and Google Sheets, in which spreadsheets are used for small-scale data storage and basic data analysis.

[0057] In some embodiments, a raw data set refers to a set of organized data, usually in the form of a table, file, database, or other structured and unstructured forms, wherein a data set usually corresponds to a data source, and a data source can correspond to multiple data sets; a raw data set contains multiple data records or entries, each record consisting of one or more fields (columns). Data sets can come from various data sources and are used in application scenarios such as data analysis, machine learning, and data mining. For different data source types, there will be corresponding types of data sets, such as database types, file types, and API types. For ease of memory, the data source can be simply compared to a database, and the data set can be compared to a table.

[0058] Optionally, the method further includes: receiving a data source connection rule sent by a user terminal, wherein the user terminal is used to display a data source interface so that the user can input the data source connection rule through the data source interface.

[0059] Combination Figure 3 As shown, the embodiment of the present disclosure provides a method for obtaining data source connection rules, including:

[0060] Step S301, displaying a data source interface to a user;

[0061] Wherein, at least a portion of the data source interface is as follows Figure 4 As shown;

[0062] Step S302: The user fills in the data source type corresponding to the original data source through the data source interface;

[0063] Among them, data source types include relational databases, NoSQL databases, data warehouses, file storage, file storage, application programming interfaces, sensors and IoT devices, spreadsheets, etc.;

[0064] Step S303: The user fills in the data source connection information according to the data source type through the data source interface;

[0065] According to different data source types, different data source connection information identifiers need to be filled in and connected to the corresponding original data source;

[0066] If the data source type is a relational database, the data source connection information includes information such as database type, host address, port, database name, account number and password;

[0067] Step S304, testing the connectivity of the original data source according to the data source type and data source connection information;

[0068] Step S305: if the test passes, generate a data source connection rule according to the data source type and the data source connection information.

[0069] Optionally, the method further includes: receiving a data set generation rule sent by a user terminal, wherein the user terminal is used to display a data set interface so that the user can input the data set generation rule through the data set interface.

[0070] Combination Figure 5 As shown, the embodiment of the present disclosure provides a method for obtaining a data set generation rule, including:

[0071] Step S501, displaying a data set interface to a user;

[0072] Wherein, at least a portion of the data set interface is as follows Figure 6 As shown;

[0073] Step S502, the user fills in the data set type through the data set interface;

[0074] The data set type includes database type, file type, Kafka type or API type, etc.

[0075] The data set type must be consistent with the data source type of the target data source;

[0076] Step S503, the user selects a target database from the original database through the data set interface;

[0077] Wherein, the target database is determined by a method of identification matching according to the user's selection;

[0078] Step S504, the user fills in the data collection information through the data set interface;

[0079] Among them, according to different data set types, you need to fill in the corresponding data collection information;

[0080] If the data set type is a database type, the data collection information includes SQL statements, etc.

[0081] If the data set type is a file type, the data collection information includes Excel files, CSV files, etc.

[0082] Among them, if the data set type is Kafka type, the data collection information package Topic, etc.

[0083] If the data set type is API type, the data collection information includes interface path information, etc.

[0084] Step S505, the user fills in the field storage information through the data set interface;

[0085] The field storage information includes field type and / or field name, and the field type includes characters, numbers or time, etc.

[0086] Optionally, the method further includes: receiving a field generation rule sent by a user terminal, wherein the user terminal is used to display a field interface so that the user can input the field generation rule through the field interface.

[0087] In some embodiments, the field name of the target field is filled in through the field interface, and the name is displayed in the field access statement with characters after MD5 processing to avoid the field name not meeting the specifications of the query engine.

[0088] Optionally, a new target field is generated according to the original field and / or the existing target field according to the field generation rule, including: if the rule type corresponding to the field generation rule is a calculated column type, the field generation rule includes a field calculation statement and a field type, the field calculation statement is executed according to the original field and / or the existing target field to obtain a calculation result, and the calculation result is verified according to the field generation rule. After the verification passes, the calculation result is stored according to the field type to obtain the target field, wherein the field calculation statement is generated based on an SQL statement.

[0089] In some embodiments, the field generation rule corresponding to the calculated column type generates the target field by writing SQL fragments. Compared with the summary column type, the calculated column type supports any SQL statement, including aggregate functions, and has higher functional flexibility.

[0090] In some embodiments, the field generation rules corresponding to the calculated column type support character, number and time types. After writing the SQL statement, the type of the newly added field can be selected according to needs. If automatic is selected, the program will obtain the field type based on the execution result of the SQL statement. Otherwise, it will determine whether the actual type executed by the SQL statement is consistent with the type selected by the user. If they are inconsistent, a forced type conversion will be performed.

[0091] In some embodiments, due to the high flexibility of SQL statements, the field generation rules corresponding to the calculated column type can be manually or automatically tested to verify whether the written SQL statement complies with the specification and can be executed correctly, and the test operation will return the actual type of the field.

[0092] Optionally, a new target field is generated according to the original field and / or the existing target field according to the field generation rule, including: if the rule type corresponding to the field generation rule is an aggregate column type, the field generation rule includes a window function, and the window function is executed according to the original field and / or the existing target field to obtain the target field output by the window function.

[0093] In some embodiments, the field generation rule corresponding to the summary column type generates the target field through the dimension, measure, sort field and summary method of the window function.

[0094] In some embodiments, the field generation rule corresponding to the summary column type uses a window function to generate target data, and the target data can only be of a numeric type, so the summary column does not need to specify a type.

[0095] In some embodiments, the field generation rules corresponding to the summary column type are strictly restricted according to the specifications of the window function and cannot be modified at will by the user. Therefore, there is no need to perform trial calculations.

[0096] Combination Figure 7As shown, the embodiment of the present disclosure provides a method for obtaining field generation rules, including:

[0097] Step S701, displaying a field interface to the user;

[0098] The field interface corresponding to the calculated column type is as follows Figure 8 As shown, the field interface corresponding to the summary column type is as follows Fig. 9 As shown;

[0099] Step S702, the user selects a rule type through the field interface;

[0100] The rule type includes a calculated column type or an aggregate column type;

[0101] Step S703, the user fills in the field name of the target field through the field interface, and jumps to step S704 and step S70;

[0102] Step S704: If the rule type is a calculated column type, the user writes an SQL statement through the field interface and fills in the field type corresponding to the target field;

[0103] Step S705, perform a trial operation according to the SQL statement and field type filled in, and if the trial operation passes, jump to step S707;

[0104] Step S706: If the rule type is a summary column type, the user fills in the dimension, measure, sort field, and summary method of the window function through the field interface, and jumps to step S707;

[0105] Step S707, saving the field generation rules;

[0106] Step S708: display the target data to the user through the field interface.

[0107] Optionally, the method also includes: establishing a linked list data structure according to the field generation rules between the target fields to obtain a field linked list composed of the target fields, including: obtaining one or more business items; classifying the target fields according to the business items to obtain global fields and private fields, wherein the private fields include project fields corresponding to the business items respectively; establishing a linked list data structure according to the field generation rules between the global fields to obtain a global field chain, and establishing a linked list data structure according to the field generation rules between the project fields to obtain project field chains corresponding to the business items respectively; linking the global field chain to each project field chain according to the field generation rules to obtain a complete field chain.

[0108] Combination Fig.10As shown, an embodiment of the present disclosure provides a complete field chain, which includes a global field chain, a project field chain 1, a project field chain 2 and a project field chain 3, wherein the global field chain includes global field A, global field B and global field C, project field chain 1 includes project field E and project field F, project field chain 2 includes project field G, project field H and project field I, project field chain 3 includes project field G and project field J, wherein the global field C has a field reference relationship with the project field E and the project field G respectively.

[0109] In some embodiments, the target fields are connected through field reference relationships to form a global field chain and multiple project field chains. The link is named With-Pipeline. The target fields in the global field chain are visible in all projects, while the target fields in the project field chain are visible only to this project. Therefore, the field name of the same target field can be the same in different project field chains, but any field name in the project field chain cannot be the same as the field name in the global field chain.

[0110] Optionally, each target field is classified according to the business project to obtain global fields and private fields, including: pre-setting the usage probability information corresponding to each original field, wherein the usage probability information includes the field usage probability corresponding to each business project; calculating according to the usage probability information corresponding to the first field to obtain the usage probability information corresponding to the second field, wherein the first field is the parent field corresponding to the second field, and the second field is any target field; arranging the field usage probabilities corresponding to the second field from large to small, and determining the first priority probability and the second priority probability; if the difference between the first priority probability and the second priority probability is greater than or equal to a preset threshold, determining the second field as the project field corresponding to the pending project, wherein the pending project is the business project corresponding to the first priority probability; if the difference between the first priority probability and the second priority probability is less than the preset threshold, determining the second field as a global field.

[0111] In some embodiments, the frequency of use of each field in each business project is counted through data analysis tools or historical query records, and the probability of use of each field in each business project is calculated accordingly. For example, the probability of use of the "order ID" field in e-commerce projects is 0.8, 0.2 in financial projects, and 0.05 in logistics projects. These data are pre-stored in the database; the hierarchical relationship between the fields is defined, for example, "user ID" is the parent field of "order ID", because orders are usually associated with users, and this hierarchical relationship is also stored in the database for subsequent calculations; for any target field (such as "delivery address"), according to the usage probability information of its parent field (such as "order ID"), a certain algorithm (such as weighted average method) is used to calculate the probability of use of the target field in each business project. Assume that the high usage probability of "order ID" in e-commerce projects affects the usage probability of its subordinate field "delivery address" in e-commerce projects; sort the calculated usage probabilities of the target fields (such as "delivery address") from large to small. Determine the first probability (first priority probability) and the second probability (second priority probability) after sorting. For example, after sorting, the probability of using "delivery address" in e-commerce projects is 0.7, which is the first-rank probability; the probability of using it in financial projects is 0.2, which is the second-rank probability; a threshold (such as 0.3) is preset to determine whether the field should be specified as a field of a certain business project or as a public field. If the difference between the first-rank probability and the second-rank probability is greater than or equal to the preset threshold (0.7-0.2=0.5>0.3), the target field (such as "delivery address") is determined as the project field of the business project (e-commerce project) corresponding to the first-rank probability. Otherwise, it is regarded as a public field, that is, a field shared by multiple business projects; according to the classification results, the field attributes in the data model are updated to mark which fields are fields of specific business projects and which are public fields. At the same time, database views can be automatically generated or updated so that users can access relevant data more conveniently.

[0112] Optionally, according to the field chain list, the access statement fragments are combined in a WITH clause expression to obtain a field access statement, including: obtaining a target project, wherein the target project is any business project; determining the current field chain corresponding to the target project from each project field chain according to the matching result between the target project and the business project, and combining the global field chain and the current field chain to obtain the field chain to be accessed; according to the link relationship between each target field in the field chain to be accessed, the access statement fragments in the field chain to be accessed are combined in sequence in a WITH clause expression, and a query main sentence is generated according to the access statement fragments to obtain a field access statement consisting of the access statement fragments and the query main sentence.

[0113] In some embodiments, when any business project needs to access a target field, a chain of fields to be accessed corresponding to the business project is obtained through matching; SQL statements corresponding to each target field on the chain of fields to be accessed are constructed in order from beginning to end according to the SQL fragments on the field reference relationship and the original field information to obtain a field access statement.

[0114] In some embodiments, the field chain to be accessed is composed of a global field chain and a project field chain. For example, if the project field chain corresponding to business project 1 is project field chain 1, and business project 1 needs to access the target field, the global field chain is combined with project field chain 1 to obtain the field chain to be accessed, and the SQL statement is completed according to the field reference relationship, so that project 1 accesses the target field on the field chain to be accessed through the SQL statement.

[0115] Optionally, the access statement fragments in the field chain to be accessed are combined in sequence using a WITH clause expression, including: if the target field does not belong to an aggregate field, then the access statement fragments corresponding to the target field are combined; if the target field belongs to an aggregate field, then when generating a query main sentence, the access statement fragments corresponding to the aggregate field are combined in the query main sentence, wherein the field generation rules corresponding to the aggregate field include aggregate functions.

[0116] In some embodiments, since aggregate functions need to be grouped when used in SQL statements, grouping will change the original number of data rows in the data set and destroy the structure of the data set. Therefore, after constructing the access statement fragments corresponding to each target field, the target field corresponding to the aggregate function is added to the query statement to achieve normal access to the target field corresponding to the aggregate function.

[0117] In some embodiments, the target fields in the field chain to be accessed include a first field, a second field, a third field and a fourth field, wherein the field generation rule corresponding to the fourth field includes an aggregation function; construct an access statement fragment corresponding to the first field, construct an access statement fragment corresponding to the second field by referencing (FROM) the access statement fragment corresponding to the first field, and construct an access statement fragment corresponding to the third field by referencing the access statement fragment corresponding to the second field; construct a query plant spacing by referencing the access statement fragment corresponding to the third field, and add the access statement fragment corresponding to the fourth field in the query main sentence.

[0118] Combination Fig.11 As shown, an embodiment of the present disclosure provides a newly added field access system for a data set, including a server end 1101 and a user terminal 1102 .

[0119] The server side 1101 is used to obtain an original data set, which includes original fields; in response to a field generation rule corresponding to the original data set, determine the parent field from the basic field, generate a target field based on the parent field according to the field generation rule, and add the target field to the basic field, wherein the basic field also includes the original field; generate a field reference relationship between the target fields according to the field generation rule; and generate a field access statement containing a WITH clause according to the field reference relationship.

[0120] The user terminal 1102 is connected to the server side, and the user terminal 1102 is used to access the target field through a field access statement.

[0121] The newly added field access system for a data set provided by the embodiment of the present disclosure is adopted. The server side uses the original field and the existing target field as the parent field, generates a new target field according to the field generation rule, and generates a field linked list according to the field generation rule between the target fields, so as to combine the access statement fragments corresponding to the target field in the WITH clause expression according to the field linked list to obtain the field access statement, and provide the target field access service to the user terminal through the field access statement. In this way, not only the organization and arrangement between the target fields is realized through the field linked list, and the number of database objects is controlled to one compared to creating multiple database tables or views to realize field expansion, which greatly simplifies the data architecture. Moreover, the field access statement corresponding to the target field is generated according to the field reference relationship through the WITH clause, and the field expansion can be realized without deactivating the data set, making the query statement more concise and efficient, thereby reducing the data set expansion steps from two aspects and improving the data set expansion efficiency.

[0122] An embodiment of the present disclosure further provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above method.

[0123] Fig.12 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Fig.12 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0124] like Fig.12As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM) 1203, such as executing the method in the above embodiment. In the random access memory 1203, various programs and data required for system operation are also stored. The central processing unit 1201, the read-only memory 1202 and the random access memory 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0125] The following components are connected to the input / output interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the input / output interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0126] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1209, and / or installed from a removable medium 1211. When the computer program is executed by a central processing unit (CPU) 1201, various functions defined in the system of the present application are executed.

[0127] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation can be changed. Parts and sub-samples of some embodiments may be included in or replace parts and sub-samples of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of a stated sub-sample, whole, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other sub-samples, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0128] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0129] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0130] The flowchart and block diagram in the accompanying drawings show the possible architecture, functions and operations of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for accessing a newly added field of a data set, characterized in that: Applied to a server side, the server side is connected to a user terminal, and the method comprises: Acquire an original data set, where the original data set includes original fields; In response to a field generation rule corresponding to the original data set, generating a new target field according to the original field and / or the existing target field in accordance with the field generation rule; Establishing a linked list data structure according to the field generation rule between the target fields, and obtaining a field linked list composed of the target fields, wherein the field linked list stores access statement fragments corresponding to the target fields; According to the field linked list, the access statement fragments are combined in a WITH clause expression to obtain a field access statement, wherein the user terminal is used to access the target field through the field access statement.

2. The method according to claim 1, characterized in that Generating a new target field according to the original field and / or the existing target field according to the field generation rule includes at least one of the following: If the rule type corresponding to the field generation rule is a calculated column type, the field generation rule includes a field calculation statement and a field type, the field calculation statement is executed according to the original field and / or the existing target field to obtain a calculation result, and the calculation result is verified according to the field generation rule. After the verification passes, the calculation result is stored according to the field type to obtain a target field, wherein the field calculation statement is generated based on an SQL statement; If the rule type corresponding to the field generation rule is a summary column type, the field generation rule includes a window function, and the window function is executed according to the original field and / or the existing target field to obtain the target field output by the window function.

3. The method according to claim 1, characterized in that A linked list data structure is established according to the field generation rules between the target fields to obtain a field linked list composed of the target fields, including: Acquire one or more business items; Classifying each of the target fields according to the business items to obtain global fields and private fields, wherein the private fields include project fields corresponding to each of the business items; A linked list data structure is established according to the field generation rules between the global fields to obtain a global field chain, and a linked list data structure is established according to the field generation rules between the project fields to obtain project field chains corresponding to the business projects; According to the field generation rule, the global field chain is linked to each of the project field chains to obtain a complete field chain.

4. The method according to claim 3, characterized in that The target fields are classified according to the business items to obtain global fields and private fields, including: Presetting usage probability information corresponding to each of the original fields, wherein the usage probability information includes field usage probabilities corresponding to each of the business items; Calculating according to the usage probability information corresponding to the first field to obtain the usage probability information corresponding to the second field, wherein the first field is the parent field corresponding to the second field, and the second field is any target field; Arrange the field usage probabilities corresponding to the second field from large to small, and determine a first priority probability and a second priority probability; If the difference between the first priority probability and the second priority probability is greater than or equal to a preset threshold, the second field is determined as a project field corresponding to a pending project, wherein the pending project is a business project corresponding to the first priority probability; If the difference between the first rank probability and the second rank probability is smaller than the preset threshold, the second field is determined as a global field.

5. The method according to claim 3, characterized in that: According to the field linked list, the access statement fragments are combined in a WITH clause expression to obtain a field access statement, including: Obtaining a target project, wherein the target project is any business project; Determine the current field chain corresponding to the target project from each of the project field chains according to the matching result between the target project and the business project, and combine the global field chain and the current field chain to obtain a field chain to be accessed; According to the link relationship between the target fields in the field chain to be accessed, the access statement fragments in the field chain to be accessed are combined in sequence in the WITH clause expression, and a query main clause is generated according to the access statement fragments to obtain a field access statement consisting of the access statement fragments and the query main clause.

6. The method according to claim 5, characterized in that The access statement fragments in the to-be-accessed field chain are sequentially combined in a WITH clause expression, including: If the target field does not belong to an aggregate field, then the access statement fragment corresponding to the target field is combined; If the target field is an aggregate field, when generating a query main sentence, an access statement fragment corresponding to the aggregate field is combined in the query main sentence, wherein the field generation rule corresponding to the aggregate field includes an aggregate function.

7. The method according to any one of claims 1 to 6, characterized in that: Get the original dataset, including: In response to a data source connection rule, the data source connection rule includes a data source type and / or data source connection information, and connecting to the original data source according to the data source connection information; In response to a data set generation rule, the data set generation rule includes data source matching information, data collection information and field storage information, matching is performed from each of the original data sources according to the data source matching information to obtain a target data source, wherein the data source matching information includes a data set type and / or a data source identifier; Collect data from the target data source according to the data collection information to obtain a target data column; The target data column is stored according to the field storage information to obtain the original field, and an original data set corresponding to the target data source is generated according to the original field, wherein the field storage information includes a field type and / or a field name.

8. The method according to claim 7, characterized in that The method further comprises at least one of the following: receiving a data source connection rule sent by the user terminal, wherein the user terminal is used to display a data source interface so that the user can input the data source connection rule through the data source interface; receiving a data set generation rule sent by the user terminal, wherein the user terminal is used to display a data set interface so that the user can input the data set generation rule through the data set interface; The field generation rule sent by the user terminal is received, wherein the user terminal is used to display a field interface so that the user can input the field generation rule through the field interface.

9. A newly added field access system for a data set, characterized in that: include: The server is used to obtain an original data set, where the original data set includes original fields; In response to the field generation rule corresponding to the original data set, a new target field is generated according to the original field and / or the existing target field according to the field generation rule; a linked list data structure is established according to the field generation rule between the target fields to obtain a field linked list composed of the target fields, wherein the field linked list stores access statement fragments corresponding to the target fields; according to the field linked list, the access statement fragments are combined in a WITH clause expression to obtain a field access statement; A user terminal is connected to the server end, and the user terminal is used to access the target field through the field access statement.

10. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method according to any one of claims 1 to 8.

Images