A multi-form data source access method

Abstract

The application relates to a multi-form data source access method, which adopts the following measures for the accessed data source: first, the field structure of the data is acquired, including the field name, type, alias, annotation and the like; second, the data is stored in the form of a file. Differentiated storage processing is carried out for different types of data source access, multi-form data source access processing, such as relational type, file type, remote file type, API type and account data type, is realized, and for the user, the operation is guided according to the page, and the user does not need to pay attention to the differences behind the implementation, so that multi-form data source access can be easily realized.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to a multi-form data source access method and system and electronic equipment. BACKGROUND

[0002] In order to realize access of different types and forms of data sources, different data sources need to be accessed, and data field structures, loading, previewing, and storing data are obtained. Subsequent data processing, analysis, and application can be performed.

[0003] In the prior art, although special data access is provided, it is generally for different database type data sources, and unified access implementation is not provided for file, API, and other types of data sources. Meanwhile, the existing data source access technology does not integrate access of different forms of data sources, and lacks some enterprise-level configuration and scheduling.

[0004] Based on the above factors, it is difficult for data enterprises to more intuitively and conveniently understand the data to be accessed, and there is no access capability for different forms and types of data sources, which further affects the ability of subsequent data analysis and application. SUMMARY

[0005] In order to solve the above problems, the present application provides a multi-form data source access method, system, and electronic equipment.

[0006] In one aspect of the present application, a multi-form data source access method is provided, comprising the following steps:

[0007] Obtaining different forms of data sources to be accessed;

[0008] Parsing the data source to obtain field information of the data source;

[0009] According to the field type of the field information, a corresponding data set record is generated in the meta database;

[0010] Writing data content in the data set into a preset feature format file to store in the form of a file, obtaining a data set format file of the data source and storing it in the meta database.

[0011] As an optional embodiment of the present application, optionally, if the data source is a relational data source, further comprising:

[0012] Defining a connector;

[0013] According to the connector model, a database account is established;

[0014] According to the database account, a connection is established, and a driver is loaded according to the connection information;

[0015] The relational data source is loaded, a connection is obtained, and a connection test is performed:

[0016] If the connection test is valid, a connection context is executed using the obtained connection and the SQL dialect used by the connector to obtain a corresponding SQL statement;

[0017] The SQL statement is visually displayed.

[0018] As an optional embodiment of the present application, optionally, when the connection context is executed using the obtained connection and the SQL dialect used by the connector, it further includes:

[0019] According to the obtained connection, a storage mapping is performed to obtain a list of stored procedures;

[0020] According to the obtained connection, a SQL query is performed to obtain a query result;

[0021] The stored procedure list and the query result are stored in the database, and a dataset record is generated in the metadata database;

[0022] The dataset record in the metadata database is processed to obtain corresponding data content, and the data content is written into a preset feature format file to be stored in the form of a file, thereby obtaining a dataset format file corresponding to the relational database type.

[0023] As an optional embodiment of the present application, optionally, when the stored procedure list and the query result are stored in the database, it further includes:

[0024] Processing data storage, selecting a direct connection or an extracted database connection mode:

[0025] If a direct connection database is selected, it is abandoned;

[0026] If an extracted database is selected, data extraction is performed on the data to be stored in the database, and distributed file operations are performed according to the extracted data content to store the data in a distributed manner.

[0027] As an optional embodiment of the present application, optionally, if the data source is a file type data source, it further includes:

[0028] Reading an uploaded file type data source;

[0029] Using a parser to parse the file type data source to obtain corresponding field information;

[0030] Detecting the field type of the field information by reading a specific number of rows, and feeding back a data table of the field type to the user;

[0031] After the user confirms the data table, a corresponding data set record is generated in the meta database;

[0032] The data content in the data set is written into a preset feature format file and stored in the meta database in the form of a file.

[0033] As an optional embodiment of the present application, optionally, if the data source is a remote file data source, further comprising:

[0034] Defining a connector;

[0035] According to the connector model, a database account is established;

[0036] According to the database account, a connection is established to load the remote file data source from a remote server;

[0037] After loading, the connection is obtained and connection testing is performed:

[0038] Reading the uploaded file type data source;

[0039] Using a parser to parse the remote file data source to obtain corresponding field information;

[0040] A data table of the field type is fed back to the user by detecting the field type of the field information in a specific row reading manner;

[0041] After the user confirms the data table, a corresponding data set record is generated in the meta database;

[0042] The data content in the data set is written into a preset feature format file and stored in the meta database in the form of a file.

[0043] As an optional embodiment of the present application, optionally, if the data source is an API data source, further comprising:

[0044] Requesting a configured http address;

[0045] If the request result is in json format, selecting the required data field from the json data;

[0046] Confirming the data field, and after confirmation, a corresponding data set record is generated in the meta database;

[0047] The data content in the data set is written into a preset feature format file and stored in the meta database in the form of a file.

[0048] As an optional embodiment of the present application, optionally, if the data source is an account data source, further comprising:

[0049] Defining a connector;

[0050] According to the connector model, a database account is established;

[0051] According to the database account, a connection is established, and an account data source is loaded from the connector according to the connection information;

[0052] The account data source is confirmed, and after confirmation, a corresponding data set record is generated in the meta database;

[0053] The data content in the data set is written into a preset feature format file and stored in the meta database in the form of a file.

[0054] Another aspect of the present application provides a system for implementing the multi-form data source access method, comprising:

[0055] An acquisition module is configured to acquire different forms of data sources to be accessed;

[0056] An analysis module is configured to analyze the data sources and acquire field information of the data sources;

[0057] A data set record module is configured to generate a corresponding data set record in a meta database according to a field type of the field information;

[0058] A storage module is configured to write data content in the data set into a preset feature format file and store the data content in the form of a file to obtain a data set format file of the data source and store the data set format file in the meta database.

[0059] Another aspect of the present application also provides an electronic device, comprising:

[0060] A processor;

[0061] A memory for storing processor executable instructions;

[0062] When the processor is configured to execute the executable instructions, the multi-form data source access method is implemented.

[0063] Technical effects of the present application:

[0064] The present application takes the following measures on the accessed data source: first, the field structure of the data is acquired, including the field name, type, alias, annotation, etc.; second, the data is stored in the form of a file. For different types of data source access, differential storage processing is performed to realize the access processing of multi-form data sources such as relational type, file type, remote file, API type and account data type. For the user, according to the page guidance operation, the difference behind the implementation does not need to be concerned, and the multi-form data source access can be easily realized.

[0065] The data source type supported by the present solution is various, in addition to the traditional database type, file type, API type, there are enterprise account data source, card-based data source, filling, etc. The process is clear, simple operation, completely visual, supports custom database type data source access.

[0066] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0068] Figure 1 The database type access process of the present application is shown as a schematic diagram;

[0069] Figure 2 The data source access general schematic diagram of the present application is shown as a schematic diagram;

[0070] Figure 3 The storage operation schematic diagram of the file type data source of the present application is shown as a schematic diagram;

[0071] Figure 4 The application schematic diagram of the electronic device of the present application is shown as a schematic diagram. DETAILED DESCRIPTION

[0072] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the drawings represent functionally the same or similar elements. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0073] The word "exemplary" is used herein in the sense of being an example, instance, or illustration. Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0074] In addition, in order to better illustrate the present disclosure, a large number of specific details are given in the specific embodiments below. Those skilled in the art should understand that the present disclosure can also be implemented without certain specific details. In some examples, means, elements and circuits that are well known to those skilled in the art are not described in detail, in order to highlight the main idea of the present disclosure.

[0075] Embodiment 1

[0076] As Figure 1 shown, in one aspect of the present application, a multi-modal data source access method is proposed, comprising the following steps:

[0077] acquire different forms of data sources to be accessed;

[0078] resolve the data sources and acquire field information of the data sources;

[0079] generate corresponding data set records in a meta database according to field types of the field information;

[0080] write data contents in the data set into a preset feature format file to store in a file form, obtain a data set format file of the data source, and store in the meta database.

[0081] The scheme first accesses all data sources, and after access, the following measures are taken on the data sources: one is to acquire field structures of data, including field names, types, aliases, annotations, etc.; two is to store and process data contents in a file form. Differentiated storage processing is performed on access of different types of data sources, and access processing of multi-form data sources such as relational type, file type, remote file, API type, and account data type is realized. For users, according to page guidance operation, the differences behind the implementation do not need to be concerned, and multi-form data source access can be easily realized.

[0082] The data sources to be accessed are databases, which are determined according to file types, such as relational data sources, which are accessed from relational databases; and file type data sources, which are accessed from other file servers or file systems.

[0083] In the resolution of field information and the storage processing of data contents, users can perform self-defined configuration of field processing information, and support self-defined database type data source access.

[0084] The feature format file is configured by a background administrator and on a background server. When data contents of a certain data source are written, a feature format file is directly called in the background.

[0085] The meta database for storing the data set format file can be determined according to user selection, which can be a database system of a third party platform or a database system of the user.

[0086] As an optional embodiment of the application, optionally, if the data source is a relational data source, the method further comprises:

[0087] defining a connector;

[0088] establishing a database account according to the connector model;

[0089] establishing a connection according to the database account, and loading a driver according to connection information;

[0090] The relational data source is loaded, a connection is obtained, and a connection test is performed:

[0091] If the connection test is valid, a connection context is executed using the obtained connection and the SQL dialect used by the connector to obtain a corresponding SQL statement.

[0092] The SQL statement is visually displayed.

[0093] For a relational data source, the following operations are mainly performed:

[0094] Step 1: Define a connector that includes connection information of a database, such as a username, a password, connection parameters, a target database, and a driver (default or custom) used.

[0095] Step 2: Create new data account information according to the connector model.

[0096] Step 3: Establish a connection according to the data account. For a relational database, connection pool management is also performed.

[0097] Step 4: Use the obtained connection and the SQL dialect used by the connector to generate different types of SQL statements, such as executing a database table list query, a table data query, a preview, a stored procedure list, details, and execution operations.

[0098] Step 5: Visualize the execution results of the SQL to facilitate users to understand the access data, including field information and data content.

[0099] Step 6: After setting the update method, data set name, and other information, a data set record is generated in the meta database, and the data of the access data source is processed. If a direct connection data set is selected, it will not be processed; otherwise, the data content is written into a specific format file for storage.

[0100] As an optional embodiment of the present application, when the connection context is executed using the obtained connection and the SQL dialect used by the connector, it also includes:

[0101] According to the obtained connection, a storage mapping is performed to obtain a stored procedure list.

[0102] According to the obtained connection, a SQL query is performed to obtain a query result.

[0103] The stored procedure list and the query result are stored in a database to generate a data set record in the meta database.

[0104] The data set record in the meta database is processed to obtain corresponding data content, and the data content is written into a preset feature format file to be stored in a file form to obtain a data set format file corresponding to the relational database type.

[0105] In combination with the accompanying Figure 1 When the verification test is performed, if it is found that the verification is valid, the field mapping is stored upward, the SQL query is performed downward, the results of the two execution modes are stored in the database, and the corresponding data set is generated.

[0106] The SQL query adopts a form query mode to obtain a query result and store the obtained query statement in the database. The query result is parsed to obtain corresponding field information, and the field information is mapped according to the field type to obtain corresponding field information. The field type of the field information is used to generate a corresponding data set record in the meta database.

[0107] When the storage mapping is performed, a storage procedure list is obtained, and a storage procedure definition is stored in the database (storage procedure storage can be visualized). The storage / query procedure can be displayed on the front-end platform of the system to obtain corresponding fields.

[0108] As an optional embodiment of the present application, when the storage procedure list and the query result are stored in the database, the following steps are further included:

[0109] The data storage is processed, and a direct connection or an extracted database connection mode is selected:

[0110] If the direct connection database is selected, the process is abandoned;

[0111] If the extracted database is selected, data extraction is performed on the data to be stored in the database, and distributed file operations are performed according to the extracted data content to store the distributed data set.

[0112] As shown in Figure 2 For the relational data source, a direct connection or an extracted record process can be selected. If the direct connection data set is selected, no processing is performed; otherwise, the data content is written into a specific format file for storage.

[0113] As an optional embodiment of the present application, if the data source is a file type data source, the following steps are further included:

[0114] The uploaded file type data source is read;

[0115] The file type data source is parsed by using a parser to obtain corresponding field information;

[0116] The field type of the field information is detected by reading a specific number of rows, and a data table of the field type is fed back to the user.

[0117] The user confirms the data table, and a corresponding data set record is generated in the meta database;

[0118] The data content in the data set is written into a preset feature format file and stored in the meta database in the form of a file.

[0119] As shown in Figure 3 the file type data source:

[0120] Step 1: Read the uploaded csv, excel, zip package of csv, etc. Type file.

[0121] Step 2: Different parsers are designed for different format files, which can parse the field information in the file data source. For the type of field, the technology of reading a specific number of rows is used to detect the field type. For the data in the file data source, a specified number of rows will be previewed according to the system configuration.

[0122] Step 3: After the user confirms the data table, set the update method, name, etc. Information, a data set record will be generated in the meta database, and the data content will be written into a specific format file.

[0123] The format file of the file type data source can be uniformly processed by the parser (such as decompression software) on the platform.

[0124] The technology of reading a specific number of rows is completed by the user.

[0125] As an optional embodiment of the present application, optionally, if the data source is a remote file data source, it further comprises:

[0126] Defining a connector;

[0127] According to the connector model, a database account is established;

[0128] According to the database account, a connection is established, and the remote file data source is loaded from the remote server;

[0129] After loading, get the connection and perform connection test:

[0130] Read the uploaded file type data source;

[0131] Use the parser to parse the remote file data source and obtain the corresponding field information;

[0132] Use the technology of reading a specific number of rows to detect the field type of the field information, and feed back a data table of the field type to the user;

[0133] The user confirms the data table, and a corresponding data set record is generated in the meta database;

[0134] Write the data content in the data set into a preset feature format file and store it in the meta database in the form of a file.

[0135] For a remote file data source:

[0136] Step 1: Define an ftp / sftp connector, including information required for establishing a connection

[0137] Step 2: Create a new data account according to the connector

[0138] Step 3: Establish a connection according to the data account and obtain files on the remote server

[0139] Step 4: The subsequent steps are basically consistent with the processing steps of the file type data source.

[0140] The connection between the platform and the source data system such as a remote server is completed by the user himself / herself to access and load.

[0141] As an optional embodiment of the present application, optionally, if the data source is an API data source, further comprising:

[0142] An http address of a request configuration;

[0143] If the request result is in json format, select the required data field from the json data;

[0144] Confirm the data field, and generate a corresponding data set record in the meta database after confirmation;

[0145] Write the data content in the data set into a preset feature format file and store it in the meta database in the form of a file.

[0146] For an API type data source:

[0147] Step 1: Request an http address of a configuration

[0148] Step 2: The result of the request is required to be in json format, and the required field is selected from the json data.

[0149] Step 3: On the basis of step 2, perform data preview, and then set update, name and other information, generate a data set record in the meta database, and write the data content into a specific format file.

[0150] As an optional embodiment of the present application, optionally, if the data source is an account data source, further comprising:

[0151] Define a connector;

[0152] According to the connector model, a database account is established;

[0153] According to the database account, a connection is established, and an account data source is loaded from the connector according to the connection information;

[0154] The account data source is confirmed, and after confirmation, a corresponding data set record is generated in the meta database;

[0155] The data content in the data set is written into a preset feature format file and stored in the meta database in the form of a file.

[0156] For account data sources:

[0157] Step 1: Support configuration of enterprise WeChat, DingTalk, OLAP, and Feishu platform connection information

[0158] Step 2: After selecting the corresponding connector, the user and user group information will be returned as the data to be accessed.

[0159] Step 3: After previewing the data, set the update method and name, a data set record will be generated in the meta database, and the data content will be written into a specific format file.

[0160] For other types of data sources:

[0161] For other types of data source access, the core implementation is still the parsing of field information and the storage and processing of data content.

[0162] Therefore, regardless of the source and format of the data source, the core implementation is still the parsing of field information and the storage and processing of data content. Two main steps are taken, one is to obtain the field structure of the data, including field name, type, alias, comment, etc.; the second is the storage and processing of data content, except for database type which can choose not to store in third-party system, other types need to store data in the form of a file. Then, for different types of data source access, differential implementation is performed.

[0163] For users, they only need to follow the page guidance operation, and do not need to pay attention to the differences behind the implementation, so they can easily realize multi-form data source access.

[0164] User-defined data configuration operations can be completed by referring to the platform's data operation components or functional modules.

[0165] Obviously, those skilled in the art should understand that all or part of the processes in the above embodiments can be instructed by a computer program to relevant hardware, and the program can be stored in a computer readable storage medium, and the program can include the processes of the above controlled embodiments when executed. Those skilled in the art can understand that all or part of the processes in the above embodiments can be instructed by a computer program to relevant hardware, and the program can be stored in a computer readable storage medium, and the program can include the processes of the above controlled embodiments when executed. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.

[0166] Embodiment 2

[0167] Based on the implementation principle of embodiment 1, another aspect of the present application provides a system for implementing the multi-form data source access method, comprising:

[0168] An acquisition module is configured to acquire different forms of data sources to be accessed.

[0169] An analysis module is configured to analyze the data sources and acquire field information of the data sources.

[0170] A data set recording module is configured to generate corresponding data set records in a meta database according to the field types of the field information.

[0171] A storage module is configured to write data contents in the data set into a preset feature format file, store the data contents in the form of a file, obtain a data set format file of the data source, and store the data set format file in the meta database.

[0172] The functions of the above modules are described in detail in embodiment 1.

[0173] The modules or steps of the present application described above can be realized by a general computing system, which can be concentrated on a single computing system or distributed on a network composed of multiple computing systems. Alternatively, they can be realized by program codes executable by a computing system, so that they can be stored in a storage system and executed by a computing system, or they can be respectively manufactured into integrated circuit modules, or multiple modules or steps among them can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0174] Embodiment 3

[0175] As Figure 4 further illustrated, another aspect of the present disclosure also provides an electronic device, comprising:

[0176] a processor;

[0177] a memory for storing processor-executable instructions;

[0178] wherein the processor is configured to implement the multi-form data source access method when executing the executable instructions.

[0179] The electronic device of the embodiments of the present disclosure comprises a processor and a memory for storing processor-executable instructions. Wherein the processor is configured to implement any one of the multi-form data source access methods when executing the executable instructions.

[0180] Here, it should be pointed out that the number of processors can be one or more. Meanwhile, the electronic device of the embodiments of the present disclosure can also comprise an input system and an output system. Wherein the processor, the memory, the input system and the output system can be connected through a bus, or can be connected through other ways, which is not specifically limited here.

[0181] The memory as a computer readable storage medium can be used to store software programs, computer executable programs and various modules, such as the programs or modules corresponding to the multi-form data source access method of the embodiments of the present disclosure. The processor executes the software programs or modules stored in the memory, thereby performing various functional applications and data processing of the electronic device.

[0182] The input system can be used to receive input numbers or signals. Wherein the signals can be key signals related to the user settings and function control of the device / terminal / server. The output system can include display devices such as display screens.

[0183] The above has described the embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles, practical applications or technical improvements of the technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A multi-form data source access method, characterized in that, The method comprises the following steps: acquiring different forms of data sources to be accessed; parsing the data sources to acquire field information of the data sources; generating corresponding data set records in a meta database according to field types of the field information; writing data contents in the data sets into preset characteristic format files to store in the form of files to obtain data set format files of the data sources and store in the meta database; if the data sources are relational data sources, the method further comprises the following steps: defining a connector; establishing a database account according to the connector model; establishing a connection according to the database account and loading a driver according to connection information; after the relational data sources are loaded, acquiring the connection and performing connection testing; if the connection testing is valid, performing connection context execution using the acquired connection and a SQL dialect used by the connector to obtain corresponding SQL statements and visually displaying the SQL statements; specifically comprising: performing storage mapping according to the acquired connection, acquiring a storage procedure list, performing SQL query according to the acquired connection, acquiring query results, storing the storage procedure list and the query results, generating data set records in the meta database, processing the data set records in the meta database to obtain corresponding data contents, and writing the data contents into preset characteristic format files to store in the form of files to obtain data set format files corresponding to the relational database type; when performing the connection testing, if it is found that the connection testing is valid, performing storage field mapping upwardly and SQL query downwardly, storing results of the two execution modes to generate corresponding data sets; performing SQL query in the form of a form query to obtain query results and store the obtained statements in the database, the query results are parsed to obtain corresponding field information, the field information is mapped according to field types to obtain corresponding field information, and field types of the field information are used to generate corresponding data set records in the meta database; when performing storage mapping, a storage procedure list is acquired, storage procedure definitions are stored in the database, and storage / query procedures are displayed on a front-end platform of the system to map corresponding fields.

2. The polymorphic data source access method of claim 1, wherein, when storing the storage procedure list and the query results in the database, the method further comprises: processing data storage and selecting direct connection or extraction of the database connection mode: if direct connection of the database is selected, the method is abandoned; if extraction of the database is selected, data extraction is performed on data to be stored in the database, and distributed file operations are performed on the extracted data contents to store the data in a distributed manner.

3. The polymorphic data source access method of claim 1, wherein, if the data sources are file type data sources, the method further comprises: reading uploaded file type data sources; using a parser to parse the file type data sources to acquire corresponding field information; using a method of reading a specific number of rows to detect field types of the field information and feeding back a data table of the field types to a user; after the user confirms the data table, corresponding data set records are generated in the meta database; data contents in the data sets are written into preset characteristic format files to store in the form of files in the meta database.

4. The polymorphic data source access method of claim 1, wherein, If the data source is a remote file data source, further comprising: defining a connector; establishing a database account according to the connector model; establishing a connection to load the remote file data source from a remote server according to the database account; after loading, obtaining a connection and performing a connection test: reading the uploaded file type data source; using a parser to parse the remote file data source and obtain corresponding field information; using a method of reading a specific number of rows to detect the field type of the field information, and feeding back a data table of the field type to the user; after the user confirms the data table, generating a corresponding data set record in the meta database; writing the data content in the data set into a preset feature format file and storing it in the meta database in the form of a file.

5. The polymorphic data source access method of claim 1, wherein, If the data source is an API data source, further comprising: requesting a configured http address; if the request result is in json format, selecting the required data field from the json data; confirming the data field, and generating a corresponding data set record in the meta database after confirmation; writing the data content in the data set into a preset feature format file and storing it in the meta database in the form of a file.

6. The polymorphic data source access method of claim 1, wherein, If the data source is an account data source, further comprising: defining a connector; establishing a database account according to the connector model; establishing a connection to load the account data source from the connector according to the connection information; confirming the account data source, and generating a corresponding data set record in the meta database after confirmation; writing the data content in the data set into a preset feature format file and storing it in the meta database in the form of a file.

7. A system for implementing the polymorphic data source access method of any of claims 1-6, wherein, comprising: an acquisition module for acquiring different forms of data sources to be accessed; a parsing module for parsing the data source and obtaining field information of the data source; a data set record module for generating a corresponding data set record in the meta database according to the field type of the field information; a storage module for writing the data content in the data set into a preset feature format file and storing it in the form of a file, obtaining a data set format file of the data source and storing it in the meta database.

8. An electronic device, comprising: comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the multi-form data source access method of any one of claims 1-6 when executing the executable instructions.

Images