Data query method and device, equipment and storage medium

Abstract

The invention discloses a data query method and device, equipment and a storage medium. The method comprises the steps that when a query request sent by a user is received, table structures in a target database are queried according to the query request, the target database comprises a plurality of data sources, and the table structures are obtained by defining data tables according to data structures of the data sources in advance; determining a target data table in each data table according to a query result, and responding to the query request according to a data source corresponding to the target data table; the table structures in the pre-constructed target database are queried according to the query request sent by the user, and the target data table is responded to the query request according to the query result, so that the problem that the user cannot effectively query a plurality of data sources due to different structures of the data sources in the database is effectively solved, and the user experience is improved. Multiple data sources can be queried at the same time, and the data query efficiency is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data query method, device, equipment and storage medium. Background Art

[0002] With the rapid development of the Internet, the amount of data generated during the development process is also growing rapidly. In order to provide better guidance to different users, it is necessary to effectively analyze the data. However, the format and size of raw data are often varied, and the compatibility between different types of structured data is poor. Data queries must be performed separately, which is very inefficient. Due to the diversification of data structures, it is currently impossible to effectively query multiple data sources at the same time, resulting in low data query efficiency.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present invention is to provide a data query method, device, equipment and storage medium, aiming to solve the technical problem that the prior art cannot effectively query multiple data sources at the same time, resulting in low data query efficiency.

[0005] To achieve the above object, the present invention provides a data query method, which comprises the following steps:

[0006] When receiving a query request sent by a user, querying each table structure in a target database according to the query request, the target database includes multiple data sources, and the table structure is obtained by pre-defining each data table according to the data structure of each data source;

[0007] Determine the target data table in each data table according to the query result;

[0008] The query request is responded to according to the data source corresponding to the target data table.

[0009] Optionally, when receiving a query request sent by a user, before querying each table structure in the target database according to the query request, the method further includes:

[0010] When receiving a data synchronization request, obtaining a data file to be synchronized according to the data synchronization request;

[0011] Identifying each data source in the data file to be synchronized;

[0012] Determine a target database according to the data synchronization request;

[0013] Each data source is stored in the target database.

[0014] Optionally, storing each data source in the target database includes:

[0015] Acquire historical query information of the target database;

[0016] Determining the data storage format of the target database according to the historical query information;

[0017] Each data source is stored in the target database according to the data storage format.

[0018] Optionally, determining the data storage format of the target database according to the historical query information includes:

[0019] Acquire historical query behavior of the target database according to the historical query information;

[0020] Determining a query format corresponding to the target database according to the historical query behavior;

[0021] The data storage format of the target database is determined according to the query format.

[0022] Optionally, storing each data source into a target database according to the data storage format includes:

[0023] Determine the table structure of the data table to which each data source belongs;

[0024] Each data source and the table structure of the data table to which each data source belongs are stored in the target database according to the data storage format.

[0025] Optionally, determining the table structure of the data table to which each data source belongs includes:

[0026] Determine the data table to which each data source belongs;

[0027] Get the data structure of each data source;

[0028] Table definition is performed on each data table according to the data structure, and the table structure of the data table to which each data source belongs is determined according to the table definition result.

[0029] Optionally, storing each data source in the target database includes:

[0030] Identify fields for each data source;

[0031] Determine the repeated fields in each data source according to the field recognition result;

[0032] Merge the repeated fields in each data source;

[0033] Each data source after field merging is stored in the target database.

[0034] Optionally, storing each data source in the target database includes:

[0035] Acquire historical query information of the target database;

[0036] Acquire historical query behavior according to the historical query information;

[0037] Determine the data arrangement order according to the historical query behavior;

[0038] Each data source is stored in the target database according to the data arrangement order.

[0039] Optionally, when receiving a query request sent by a user, querying each table structure in the target database according to the query request includes:

[0040] When receiving a query request sent by a user, obtaining the query logic of the query request;

[0041] Verifying the query request according to the query logic;

[0042] When the verification result meets the preset requirements, each table structure in the target database is queried according to the query request.

[0043] Optionally, verifying the query request according to the query logic includes:

[0044] Determine the to-be-queried field of the query request according to the query logic;

[0045] Traversing the target database according to the field to be queried;

[0046] It is determined whether the field to be queried exists according to the traversal result to complete the verification of the query request.

[0047] In addition, to achieve the above object, the present invention also provides a data query device, the data query device comprising:

[0048] A request receiving module, configured to query each table structure in a target database according to the query request when receiving a query request sent by a user, wherein the target database includes multiple data sources, and the table structure is obtained by pre-defining each data table according to the data structure of each data source;

[0049] A data identification module, used to determine a target data table in each data table according to the query result;

[0050] The query response module is used to respond to the query request according to the data source corresponding to the target data table.

[0051] Optionally, the data query device further includes:

[0052] The data storage module is used to obtain the data file to be synchronized according to the data synchronization request when receiving the data synchronization request; identify each data source in the data file to be synchronized; determine the target database according to the data synchronization request; and store each data source in the target database.

[0053] Optionally, the data storage module is further used to obtain historical query information of the target database; determine the data storage format of the target database according to the historical query information; and store each data source in the target database according to the data storage format.

[0054] Optionally, the data storage module is further used to obtain historical query behavior of the target database according to the historical query information; determine the query format corresponding to the target database according to the historical query behavior; and determine the data storage format of the target database according to the query format.

[0055] Optionally, the data storage module is further used to determine the table structure of the data table to which each data source belongs; and store each data source and the table structure of the data table to which each data source belongs in the target database according to the data storage format.

[0056] Optionally, the data storage module is further used to determine the data table to which each data source belongs; obtain the data structure of each data source; define each data table according to the data structure, and determine the table structure of the data table to which each data source belongs according to the table definition result.

[0057] Optionally, the data storage module is further used to identify fields in each data source; determine repeated fields in each data source based on the field identification results; merge the repeated fields in each data source; and store each data source after the field merging in the target database.

[0058] Optionally, the data storage module is further used to obtain historical query information of the target database; obtain historical query behavior based on the historical query information; determine the data arrangement order based on the historical query behavior; and store each data source to the target database according to the data arrangement order.

[0059] In addition, to achieve the above-mentioned purpose, the present invention also proposes a data query device, which includes: a memory, a processor, and a data query program stored in the memory and executable on the processor, wherein the data query program is configured to implement the steps of the data query method described above.

[0060] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which a data query program is stored, and when the data query program is executed by a processor, the steps of the data query method described above are implemented.

[0061] The present invention, when receiving a query request sent by a user, queries each table structure in a target database according to the query request, the target database contains multiple data sources, the table structure is obtained by pre-defining each data table according to the data structure of each data source, determines the target data table in each data table according to the query result, and responds to the query request according to the data source corresponding to the target data table; because the present invention pre-defines each data table in the target database, thereby obtaining the table structure of each data table, and then queries the table structure in the target database according to the query request sent by the user, obtains the target data table according to the query structure, and then responds to the query request according to the data source corresponding to the target data table, thereby effectively solving the problem that the structure of each data source in the database is different, resulting in the user being unable to effectively query multiple data sources, and realizing simultaneous query of multiple data sources, meeting the user's needs for querying multiple data sources at one time, effectively improving the data query efficiency, and also improving the user's efficiency in analyzing and processing data. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a structural diagram of a data query device in a hardware operating environment involved in an embodiment of the present invention;

[0063] Figure 2 This is a flow chart of a first embodiment of a data query method of the present invention;

[0064] Figure 3 This is a flow chart of a second embodiment of a data query method of the present invention;

[0065] Figure 4 This is a flow chart of a third embodiment of a data query method of the present invention;

[0066] Figure 5 It is a structural block diagram of the first embodiment of the data query device of the present invention.

[0067] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0068] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0069] Reference Figure 1 , Figure 1 The present invention is a schematic diagram of the data query device structure of the hardware operating environment involved in the embodiment of the present invention.

[0070] like Figure 1 As shown, the data query device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0071] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the data query device, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0072] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a data query program.

[0073] exist Figure 1 In the data query device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the data query device of the present invention can be set in the data query device, and the data query device calls the data query program stored in the memory 1005 through the processor 1001, and executes the data query method provided by the embodiment of the present invention.

[0074] The embodiment of the present invention provides a data query method, referring to Figure 2 , Figure 2 The figure is a flow chart of a first embodiment of a data query method of the present invention.

[0075] In this embodiment, the data query method includes the following steps:

[0076] Step S10: when receiving a query request sent by a user, query each table structure in the target database according to the query request, the target database includes multiple data sources, and the table structure is pre-defined based on the data structure of each data source.

[0077] It should be understood that the executor of the method of this embodiment can be a data query device with data processing, network communication and program running functions, such as a computer, or other devices or equipment that can achieve the same or similar functions. The above-mentioned data query device is used as an example for explanation here.

[0078] It should be noted that the query request may be a data query request sent by a user, and the query request may include the type of data and data content that the user needs to query. The above-mentioned target database may be a database pre-built by the data query device that stores one or more data sources, and the target database may be the database specified by the query request. The above-mentioned data source may be a table in the traditional sense as a data source, that is, a series of structured data with the same data structure (DataSource Schema), which may be data from the same database table, or a network log in the same format, etc.

[0079] The above data table can be a table composed of one or more data sources. The data sources in the data table have common fields, and the field types, names, and maximum lengths are consistent. Of course, it is also acceptable to have no common fields. The above requirements can be converted into: there cannot be any field conflicts between this group of data sources, that is, there are no fields with the same name but different types or maximum lengths between any two data sources. The data source and the table can have the same name, because at the query level, only the table is used as the object of the query statement, and the data source name can only be used as a supplementary field. In particular, when a data table contains only one data source, the data source and the data table are equivalent.

[0080] The above table structure can be the structure (schema) of the data table, and the table structure can be the sum of the structures (Data Source Schema) of the data sources contained in the data table. The data query device obtains the table structure by unioning the fields of different data sources and removing duplicates. For any data source, the table structure contains the data source structure. If the data table contains only one data source, the table structure and the data source structure are truly contained, or equivalent. The definition of the data source, table, and table structure can define the table. Since the definition of each data source has been defined by the upstream data source, once the table definition is completed, the table structure is also completed.

[0081] It should be understood that a query request can start with a query statement, which can be a query on a table or some tables. The table here is a set of data sources defined above. If a field does not exist on a data source, it is considered that the field is null for the data source. The difference between the table structure and the data source structure is the column that needs to be filled with Null, and feedback is given to the user.

[0082] Furthermore, in order to verify the query request sent by the user, the above step S10 may include:

[0083] When receiving a query request sent by a user, obtaining the query logic of the query request;

[0084] Verifying the query request according to the query logic;

[0085] When the verification result meets the preset requirements, each table structure in the target database is queried according to the query request.

[0086] It should be understood that when the data query device receives a query request, it will verify the query logic in the query request. If there is a problem, it will directly return an error without continuing to the next step. The verification is mainly field verification, including whether the field to be queried actually exists, whether the type is wrong, etc. This requires the table structure previously saved in the database. Of course, the saved table structure will also be exposed to the user in some way to let the user know which fields can be queried.

[0087] Further, in order to verify the accuracy, the verification of the query request according to the query logic may include:

[0088] Determine the to-be-queried field of the query request according to the query logic;

[0089] Traversing the target database according to the field to be queried;

[0090] It is determined whether the field to be queried exists according to the traversal result to complete the verification of the query request.

[0091] It should be noted that the field to be queried may be a data field to be queried in a query request sent by a user.

[0092] It should be understood that the data query device determines the field to be queried of the query request according to the query logic, identifies the field type of the field to be queried, traverses the table structures in the target database according to the field type, and determines whether the field to be queried exists according to the traversal result. If it exists, it responds to the user's query request according to the traversed table structure. If it does not exist, it returns an invalid value, that is, returns Null, to complete the verification of the query request.

[0093] Step S20: Determine the target data table among the data tables according to the query result.

[0094] It should be noted that the query result may be a result of a table structure queried based on a user's query request. The above target data table may be a data table whose table structure is consistent with the structure queried in the query request.

[0095] It should be understood that the data query device of this embodiment determines the target structure that the user needs to query based on the query request, queries the various table structures in the target database based on the target structure, finds the target table structure consistent with the target structure from the table structure, and determines the target data table in each data table based on the target table structure.

[0096] Step S30: responding to the query request according to the data source corresponding to the target data table.

[0097] It should be noted that the data query device parses the query request sent by the user, determines the target field that the user needs to query according to the query request, determines the data type to be queried according to the target field, determines the target structure to be queried according to the data type, and determines the target data table in the target database according to the target structure. Since the target data table contains one or more data sources, and the data sources have common fields, each data source in the target data table is the data source required for the user's query request, and the data source corresponding to the target data table responds to the user's query request, thereby realizing simultaneous query of multiple data sources and meeting the user's needs to query multiple data sources at one time.

[0098] It should be understood that the data query device can use big data processing engines such as Apache Flink as query engines when querying. Although Apache Flink itself supports SQL queries on files, the number of fields in specific files is not the same, so directly using the existing application programming interface (API) cannot directly achieve the purpose. The data query device transforms the API, and returns Null if a field that the file itself does not have is queried. And using Flink's API interface, the data source is deeply customized to support simultaneous scanning of multiple files with different structures. Although in a large category, each type of data source file is heterogeneous, the data query device "treats them equally" and mistakenly regards them as isomorphic, but some fields are empty. In this way, there is no priority or particularity between each file. Different treatment is only carried out when the user explicitly specifies certain conditions. For example, if the most recent data is to be queried, the data query device will arrange these files according to the time of occurrence, thereby realizing the pluggable nature of the data source. The connected data sources can be increased or decreased at will, and the fields of each data source can be modified at any time, thus realizing the decoupling between the query layer and the data generation source.

[0099] This embodiment, when receiving a query request sent by a user, queries each table structure in a target database according to the query request, the target database includes multiple data sources, the table structure is obtained by pre-defining each data table according to the data structure of each data source, determines the target data table in each data table according to the query result, and responds to the query request according to the data source corresponding to the target data table; because this embodiment pre-defines each data table in the target database to obtain the table structure of each data table, and then queries the table structure in the target database according to the query request sent by the user, obtains the target data table according to the query structure, and then responds to the query request according to the data source corresponding to the target data table, the problem that the structure of each data source in the database is different, resulting in the user being unable to effectively query multiple data sources, is effectively solved, and multiple data sources are queried simultaneously, meeting the user's needs for querying multiple data sources at one time, effectively improving the data query efficiency, and also improving the user's efficiency in analyzing and processing data.

[0100] refer to Figure 3 , Figure 3 The figure is a flow chart of a second embodiment of a data query method of the present invention.

[0101] Based on the first embodiment above, in this embodiment, before step S10, the following steps are included:

[0102] Step S11: when a data synchronization request is received, a data file to be synchronized is obtained according to the data synchronization request.

[0103] It should be noted that the data synchronization request may be a request sent by a user to synchronize data to a target database. The data file to be synchronized may be a data file storing raw data, and the raw data may be log data collected by a data collector, or may be an individual behavior log of a person, device, etc. required by a user for data analysis.

[0104] Step S12: Identify each data source in the data file to be synchronized.

[0105] It should be noted that the data source can be the behavior logs of individuals such as personnel and equipment that users need to use when conducting data analysis (such as network security data analysis). The log format can be pre-defined by the user, and the type of log varies according to the actual situation of different customers. The logs in the data files to be synchronized can be divided into two categories, including events (EVENT) and alarms (ALARM). Except for a small number of users who query a specific type of log, most of the other queries are based on the entire category.

[0106] Step S13: Determine the target database according to the data synchronization request.

[0107] It should be noted that, since different data sources require different databases, the data query device in this embodiment determines the data structure of each data source in the data file to be synchronized according to the data synchronization request, and allocates each data source to the corresponding database according to the data structure.

[0108] Step S14: storing each data source in the target database.

[0109] Furthermore, in order to improve storage efficiency, the above step S14 may include:

[0110] Step S141: Acquire historical query information of the target database;

[0111] Step S142: determining the data storage format of the target database according to the historical query information;

[0112] Step S143: storing each data source into the target database according to the data storage format.

[0113] It should be noted that the historical query information may be the historical records recorded by the user when querying the target database, and the historical query information may include the historical query method and historical query format of the user in the historical query behavior. The above data storage format may be the storage format of the target database to store data, for example, the data storage format may include column format storage and row format storage, etc.

[0114] It should be understood that the data query device of this embodiment obtains historical query information of the target database, identifies the user's historical query behavior based on the historical query behavior, determines the data storage format of the target database based on the user's historical query behavior, and stores each data source in the target database according to the data storage format. It can also identify the storage format of the data already stored in the target database and determine the data storage format of the target database based on the storage format of the stored data.

[0115] Furthermore, in order to improve the query efficiency of subsequent users, the above step S142 may include:

[0116] Step S1421: Acquire historical query behavior of the target database according to the historical query information;

[0117] Step S1422: determining a query format corresponding to the target database according to the historical query behavior;

[0118] Step S1423: Determine the data storage format of the target database according to the query format.

[0119] It should be noted that the historical query behavior may be a historical behavior log recorded by a data query device when a user queries a target database for data, and the query format may be a query format corresponding to the user's historical query behavior.

[0120] In a specific implementation, the data query device can store data according to the data source as the smallest classification unit. This is because the structure of the data generated by the same data source must be consistent. For example, when the user's data query format is a column query, in order to better meet the filtering and querying of columns, a column format storage format can be used. Common solutions include Apache Parquet, Apache ORC, etc. When storing data, it can be stored according to the original structure, that is, as many fields as there are, without considering the influence of the table structure.

[0121] Furthermore, in order to display the data types stored in the target database to the user, the above step S143 may include:

[0122] Step S1431: determine the table structure of the data table to which each data source belongs;

[0123] Step S1432: storing each data source and the table structure of the data table to which each data source belongs in the target database according to the data storage format.

[0124] It should be noted that the table structure can be the structure (schema) of the data table, and the table structure can be the sum of the structures (Data Source Schema) of the data sources contained in the data table. The data query device obtains the table structure by unioning the fields of different data sources and removing duplicates. For any data source, the table structure contains the data source structure. If the data table contains only one data source, the table structure and the data source structure are truly contained, or equivalent. The definition of the data source, table, and table structure can define the table. Since the definition of each data source has been defined by the upstream data source, once the table definition is completed, the table structure is also completed.

[0125] Furthermore, in order to accurately determine the table structure of the data table to which each data source belongs, the above step S1431 may include:

[0126] Determine the data table to which each data source belongs;

[0127] Get the data structure of each data source;

[0128] Table definition is performed on each data table according to the data structure, and the table structure of the data table to which each data source belongs is determined according to the table definition result.

[0129] It should be understood that the data query device traverses each data source in the target database, determines the field type contained in each data source according to the traversal results, classifies the fields of each data source according to the field type, and defines the data table according to the classification results, thereby determining the data table to which each data source belongs based on the data table definition. Since the definition of each data source has already been defined by the upstream data source, once the table definition is completed, the table structure is also completed.

[0130] For example, the data query device determines each storage path in the target database based on the data storage format of the target database, traverses the data according to each storage path, determines that data source A, data source B, data source C and data source D are stored in the target database according to the traversal results, identifies the fields contained in each data source, thereby determining the field type in each data source, and classifies the data sources based on the field types, that is, determines that the field types of data source B and data source D are consistent, and the field types of data source A and data source C are consistent according to the field types, and therefore defines the data table based on the classification results, determines data table 1 based on data source A and data source C, determines data table 2 based on data source B and data source D, and determines the table structures of data table 1 and data table 2 according to the data structures of data source A, data source B, data source C and data source D, respectively.

[0131] Further, in order to improve data storage efficiency, the above step S14 may include:

[0132] Acquire historical query information of the target database;

[0133] Acquire historical query behavior according to the historical query information;

[0134] Determine the data arrangement order according to the historical query behavior;

[0135] Each data source is stored in the target database according to the data arrangement order.

[0136] It should be noted that historical query information can be the historical behavior record information of the user in the target database. The access behavior may include data query, data synchronization, data migration, data access, etc. When the data query device detects that the target database has a user access, it will record the user's behavior and generate the user's historical behavior log based on the record results. The historical query behavior can be the query logic of the user's historical query data, for example, the historical query behavior can be the user's historical query format and query data type.

[0137] It should be understood that the data query device obtains the historical behavior log of the target database, determines the user's historical query information based on the historical behavior log, determines the user's historical query behavior based on the historical query information, determines the user's query format and query habits based on the historical query behavior. For example, the query format can be a columnar query format, determines the user's query data behavior priority based on the query habits, determines the data arrangement order based on the behavior priority, and stores each data source in the target database according to the data arrangement order.

[0138] In the specific implementation, when the data query device is storing data, there is no priority or particularity between each file. Differentiation is only performed when the user explicitly specifies certain conditions. For example, if the most recent data is to be queried, the data query device will arrange the files according to the time of occurrence.

[0139] This embodiment obtains the data file to be synchronized according to the data synchronization request when receiving the data synchronization request, identifies each data source in the data file to be synchronized, determines the target database according to the data synchronization request, and stores each data source in the target database; because this embodiment identifies the data file to be synchronized when receiving the data synchronization request, and allocates a database to each data source in the data file to be synchronized according to the identification result, and stores each data source in the target database, the rationality of the data source storage is ensured, and the query efficiency for subsequent users to perform data queries is improved.

[0140] refer to Figure 4 , Figure 4 The figure is a flow chart of a third embodiment of a data query method of the present invention.

[0141] Based on the above second embodiment, in this embodiment, step S14 includes:

[0142] Step S15: Identify the fields of each data source;

[0143] Step S16: determining repeated fields in each data source according to the field recognition result;

[0144] Step S17: Merge the repeated fields in each data source;

[0145] Step S18: storing each data source after field merging into the target database.

[0146] It should be noted that repeated fields can be fields with the same name or the same type. For example, if the "name" field exists in both data source A and data source B, then data source A has a repeated field in data source B.

[0147] It should be understood that when the data query device performs data synchronization, the data file can be saved in the form of Apache Parquet. It should be noted that since the format of each type of log can also change all the time, the data stored in the same file may also have different fields. The data query device uses the method of deduplication and merging to deal with this problem. After each deduplication and merging, the result is saved in the database.

[0148] In a specific implementation, the data query device determines each storage path in the target database based on the data storage format of the target database, traverses the data according to each storage path, determines each data source stored in the target database according to the traversal result, identifies the field of each data source, performs name comparison and type comparison on the fields in each data source according to the field identification result, thereby determining the repeated fields in each data source according to the name comparison result and the type comparison result, deduplicates the repeated fields and merges them, retains only a group of fields of the same type or name, and stores each data source after the field is merged in the target database.

[0149] For example, the data query device determines each storage path in the target database based on the data storage format of the target database, traverses the data according to each storage path, determines the data source C, data source D, data source E and data source F stored in the target database according to the traversal results, and determines the field type and field name of each field in each data source by identifying the field of each data source. According to the field type and field name, it is determined that the field name exists in the data source C, data source D, and data source E. Therefore, the repeated fields in the data source C, data source D, and data source E are merged, and the data source C, data source D, and data source E after the field merger are stored in the target database.

[0150] This embodiment performs field recognition on each data source, determines the repeated fields in each data source according to the field recognition result, merges the repeated fields in each data source, and stores each data source after the field merger in the target database; because this embodiment performs field recognition on each data source, determines the repeated fields in each data source according to the field recognition result, thereby realizing the screening of repeated data, merges the repeated fields in each data source, thereby realizing the de-duplication and merging of each data source, and stores each data source after the field merger in the target database, the problem of inconsistent fields caused by changes in the field format in the data source is effectively solved, and the stability of the data is ensured.

[0151] In addition, an embodiment of the present invention further provides a storage medium, on which a data query program is stored. When the data query program is executed by a processor, the steps of the data query method described above are implemented.

[0152] Since the storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.

[0153] Reference Figure 5 , Figure 5 It is a structural block diagram of the first embodiment of the data query device of the present invention.

[0154] like Figure 5 As shown, the data query device proposed in the embodiment of the present invention includes:

[0155] The request receiving module 10 is used to query each table structure in the target database according to the query request when receiving the query request sent by the user, wherein the target database includes multiple data sources, and the table structure is obtained by pre-defining each data table according to the data structure of each data source;

[0156] A data identification module 20, used to determine a target data table among various data tables according to the query result;

[0157] The query response module 30 is used to respond to the query request according to the data source corresponding to the target data table.

[0158] Furthermore, the data query device also includes:

[0159] The data storage module 40 is used to obtain the data file to be synchronized according to the data synchronization request when receiving the data synchronization request; identify each data source in the data file to be synchronized; determine the target database according to the data synchronization request; and store each data source in the target database.

[0160] Furthermore, the data storage module 40 is also used to obtain historical query information of the target database; determine the data storage format of the target database according to the historical query information; and store each data source in the target database according to the data storage format.

[0161] Furthermore, the data storage module 40 is also used to obtain the historical query behavior of the target database according to the historical query information; determine the query format corresponding to the target database according to the historical query behavior; and determine the data storage format of the target database according to the query format.

[0162] Furthermore, the data storage module 40 is also used to determine the table structure of the data table to which each data source belongs; and store each data source and the table structure of the data table to which each data source belongs in the target database according to the data storage format.

[0163] Furthermore, the data storage module 40 is also used to determine the data table to which each data source belongs; obtain the data structure of each data source; define each data table according to the data structure, and determine the table structure of the data table to which each data source belongs according to the table definition result.

[0164] Furthermore, the data storage module 40 is also used to identify fields of each data source; determine repeated fields in each data source according to the field identification result; merge the repeated fields in each data source; and store each data source after the field merging in the target database.

[0165] Furthermore, the data storage module 40 is also used to obtain historical query information of the target database; obtain historical query behavior based on the historical query information; determine the data arrangement order based on the historical query behavior; and store each data source to the target database according to the data arrangement order.

[0166] Furthermore, the request receiving module 10 is also used to obtain the query logic of the query request when receiving the query request sent by the user; verify the query request according to the query logic; when the verification result meets the preset requirements, query each table structure in the target database according to the query request.

[0167] Furthermore, the request receiving module 10 is also used to determine the to-be-queried field of the query request according to the query logic; traverse the target database according to the to-be-queried field; and determine whether the to-be-queried field exists according to the traversal result to complete the verification of the query request.

[0168] This embodiment, when receiving a query request sent by a user, queries each table structure in a target database according to the query request, the target database includes multiple data sources, the table structure is obtained by pre-defining each data table according to the data structure of each data source, determines the target data table in each data table according to the query result, and responds to the query request according to the data source corresponding to the target data table; because this embodiment pre-defines each data table in the target database to obtain the table structure of each data table, and then queries the table structure in the target database according to the query request sent by the user, obtains the target data table according to the query structure, and then responds to the query request according to the data source corresponding to the target data table, the problem that the structure of each data source in the database is different, resulting in the user being unable to effectively query multiple data sources, is effectively solved, and multiple data sources are queried simultaneously, meeting the user's needs for querying multiple data sources at one time, effectively improving the data query efficiency, and also improving the user's efficiency in analyzing and processing data.

[0169] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0170] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0171] In addition, for technical details that are not described in detail in this embodiment, reference can be made to the data query method provided in any embodiment of the present invention, and will not be repeated here.

[0172] The present application discloses A1, a data query method, the data query method comprising:

[0173] When receiving a query request sent by a user, querying each table structure in a target database according to the query request, the target database includes multiple data sources, and the table structure is obtained by pre-defining each data table according to the data structure of each data source;

[0174] Determine the target data table in each data table according to the query result;

[0175] The query request is responded to according to the data source corresponding to the target data table.

[0176] A2. The data query method as described in A1, wherein upon receiving a query request sent by a user, before querying each table structure in the target database according to the query request, the method further comprises:

[0177] When receiving a data synchronization request, obtaining a data file to be synchronized according to the data synchronization request;

[0178] Identifying each data source in the data file to be synchronized;

[0179] Determine a target database according to the data synchronization request;

[0180] Each data source is stored in the target database.

[0181] A3. The data query method as described in A2, wherein storing each data source in the target database comprises:

[0182] Acquire historical query information of the target database;

[0183] Determining the data storage format of the target database according to the historical query information;

[0184] Each data source is stored in the target database according to the data storage format.

[0185] A4. The data query method as described in A3, wherein determining the data storage format of the target database according to the historical query information comprises:

[0186] Acquire historical query behavior of the target database according to the historical query information;

[0187] Determining a query format corresponding to the target database according to the historical query behavior;

[0188] The data storage format of the target database is determined according to the query format.

[0189] A5. The data query method as described in A4, wherein storing each data source into the target database according to the data storage format comprises:

[0190] Determine the table structure of the data table to which each data source belongs;

[0191] Each data source and the table structure of the data table to which each data source belongs are stored in the target database according to the data storage format.

[0192] A6. The data query method as described in A5, wherein determining the table structure of the data table to which each data source belongs comprises:

[0193] Determine the data table to which each data source belongs;

[0194] Get the data structure of each data source;

[0195] Table definition is performed on each data table according to the data structure, and the table structure of the data table to which each data source belongs is determined according to the table definition result.

[0196] A7. The data query method as described in A2, wherein storing each data source in the target database comprises:

[0197] Identify fields for each data source;

[0198] Determine the repeated fields in each data source according to the field recognition result;

[0199] Merge the repeated fields in each data source;

[0200] Each data source after field merging is stored in the target database.

[0201] A8. The data query method as described in A2, wherein storing each data source in the target database comprises:

[0202] Acquire historical query information of the target database;

[0203] Acquire historical query behavior according to the historical query information;

[0204] Determine the data arrangement order according to the historical query behavior;

[0205] Each data source is stored in the target database according to the data arrangement order.

[0206] A9. The data query method as described in any one of A1 to A8, wherein upon receiving a query request sent by a user, querying each table structure in the target database according to the query request comprises:

[0207] When receiving a query request sent by a user, obtaining the query logic of the query request;

[0208] Verifying the query request according to the query logic;

[0209] When the verification result meets the preset requirements, each table structure in the target database is queried according to the query request.

[0210] A10. The data query method as described in A9, wherein verifying the query request according to the query logic comprises:

[0211] Determine the to-be-queried field of the query request according to the query logic;

[0212] Traversing the target database according to the field to be queried;

[0213] It is determined whether the field to be queried exists according to the traversal result to complete the verification of the query request.

[0214] The present application also discloses B11, a data query device, the data query device comprising:

[0215] A request receiving module, configured to query each table structure in a target database according to the query request when receiving a query request sent by a user, wherein the target database includes multiple data sources, and the table structure is obtained by pre-defining each data table according to the data structure of each data source;

[0216] A data identification module, used to determine a target data table in each data table according to the query result;

[0217] The query response module is used to respond to the query request according to the data source corresponding to the target data table.

[0218] B12. The data query device as described in B11, further comprising:

[0219] The data storage module is used to obtain the data file to be synchronized according to the data synchronization request when receiving the data synchronization request; identify each data source in the data file to be synchronized; determine the target database according to the data synchronization request; and store each data source in the target database.

[0220] B13. In the data query device as described in B12, the data storage module is further used to obtain historical query information of the target database; determine the data storage format of the target database according to the historical query information; and store each data source in the target database according to the data storage format.

[0221] B14. In the data query device as described in B13, the data storage module is further used to obtain the historical query behavior of the target database according to the historical query information; determine the query format corresponding to the target database according to the historical query behavior; and determine the data storage format of the target database according to the query format.

[0222] B15. In the data query device as described in B14, the data storage module is further used to determine the table structure of the data table to which each data source belongs; and store each data source and the table structure of the data table to which each data source belongs in the target database according to the data storage format.

[0223] B16. In the data query device as described in B15, the data storage module is further used to determine the data table to which each data source belongs; obtain the data structure of each data source; define each data table according to the data structure, and determine the table structure of the data table to which each data source belongs according to the table definition result.

[0224] B17. In the data query device as described in B12, the data storage module is further used to identify fields of each data source; determine repeated fields in each data source according to the field identification result; merge the repeated fields in each data source; and store each data source after the field merging in the target database.

[0225] B18. In the data query device as described in B12, the data storage module is also used to obtain historical query information of the target database; obtain historical query behavior based on the historical query information; determine the data arrangement order based on the historical query behavior; and store each data source in the target database according to the data arrangement order.

[0226] The present application also discloses C19, a data query device, which includes: a memory, a processor, and a data query program stored in the memory and executable on the processor, wherein the data query program is configured to implement the data query method as described above.

[0227] The present application also discloses D20, a storage medium, on which a data query program is stored, and when the data query program is executed by a processor, the data query method as described above is implemented.

[0228] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0229] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0230] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0231] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data query method, characterized in that: The data query method comprises: When receiving a query request sent by a user, querying each table structure in a target database according to the query request, the target database includes multiple data sources, and the table structure is obtained by pre-defining each data table according to the data structure of each data source; Determine the target data table in each data table according to the query result; The query request is responded to according to the data source corresponding to the target data table.

2. The data query method according to claim 1, characterized in that: When receiving a query request sent by a user, before querying each table structure in the target database according to the query request, the method further includes: When receiving a data synchronization request, obtaining a data file to be synchronized according to the data synchronization request; Identifying each data source in the data file to be synchronized; Determine a target database according to the data synchronization request; Each data source is stored in the target database.

3. The data query method according to claim 2, characterized in that: The storing of each data source into the target database comprises: Acquire historical query information of the target database; Determining the data storage format of the target database according to the historical query information; Each data source is stored in the target database according to the data storage format.

4. The data query method according to claim 3, characterized in that: Determining the data storage format of the target database according to the historical query information includes: Acquire historical query behavior of the target database according to the historical query information; Determining a query format corresponding to the target database according to the historical query behavior; The data storage format of the target database is determined according to the query format.

5. The data query method according to claim 4, characterized in that: The storing each data source into the target database according to the data storage format includes: Determine the table structure of the data table to which each data source belongs; Each data source and the table structure of the data table to which each data source belongs are stored in the target database according to the data storage format.

6. The data query method according to claim 5, characterized in that: Determining the table structure of the data table to which each data source belongs includes: Determine the data table to which each data source belongs; Get the data structure of each data source; Table definition is performed on each data table according to the data structure, and the table structure of the data table to which each data source belongs is determined according to the table definition result.

7. The data query method according to claim 2, characterized in that: The storing of each data source into the target database comprises: Identify fields for each data source; Determine the repeated fields in each data source according to the field recognition result; Merge the repeated fields in each data source; Each data source after field merging is stored in the target database.

8. A data query device, characterized in that: The data query device comprises: A request receiving module, configured to query each table structure in a target database according to the query request when receiving a query request sent by a user, wherein the target database includes multiple data sources, and the table structure is obtained by pre-defining each data table according to the data structure of each data source; A data identification module, used to determine a target data table in each data table according to the query result; The query response module is used to respond to the query request according to the data source corresponding to the target data table.

9. A data query device, characterized in that: The data query device comprises: a memory, a processor, and a data query program stored in the memory and executable on the processor, wherein the data query program is configured to implement the data query method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a data query program, and when the data query program is executed by the processor, the data query method according to any one of claims 1 to 7 is implemented.

Images