Data processing method, device, electronic device and computer storage medium

Abstract

The embodiment of the present invention provides a data processing method, device, electronic device and computer storage medium. The method includes: obtaining a creation instruction of a target data table, obtaining setting dynamic field information from the creation instruction, wherein the setting dynamic field information is used to indicate the definition rule of the dynamic field in the target data table; according to the definition rule indicated in the setting dynamic field information, creating the target data table, the target data table includes the dynamic field defined by the definition rule, and saving the setting dynamic field information to the metadata of the target data table. Through the embodiment of the present invention, ease of use and convenience can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a data processing method, device, electronic device and computer storage medium. Background Art

[0002] With the advent of the big data era, the amount of data that needs to be stored has increased dramatically. In order to meet the new data storage, management and usage requirements, people use data lakes to store and manage data. In the data lake usage scenario, users also need to define databases and data tables in the data lake, and map the database or data table to various data sources through path, protocol and other information. When querying the data table, the data source is found through the mapping relationship, and the data is read from it for analysis.

[0003] Whether it is a traditional database or a data lake, when defining a data table, all fields in the data table need to be completely and clearly defined, so that various processing can be performed on the fields in the data table during subsequent query analysis.

[0004] However, in data lake usage scenarios, the field information in the real data source corresponding to the data table often changes or is even unknown. For example, in KV (key-value) and wide table storage databases such as HBase, BigTable, and TableStore, only the primary key field is fixed, while non-primary key fields are allowed to change.

[0005] In view of this situation, in order to ensure that SQL can be used normally to process data in the database, it is necessary to repeatedly change the data table structure and update the metadata of the data table through methods such as DDL (Data Definition Language) every time a field in the data table changes. The problem with this method is that DDL is used to process the fields in the data table, such as adding fields, deleting fields, etc., only after the data content changes. This method of frequently changing the data table structure using DDL is difficult because the data table cannot be processed normally without changing the table structure and metadata, resulting in cumbersome processing of the metadata corresponding to the data table stored in the database, poor usability, and cumbersome operation. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a data processing solution to solve some or all of the above problems.

[0007] According to a first aspect of an embodiment of the present invention, a data processing method is provided, comprising: obtaining a creation instruction for a target data table, obtaining setting dynamic field information from the creation instruction, wherein the setting dynamic field information is used to indicate a definition rule for a dynamic field in the target data table; creating the target data table according to the definition rule indicated in the setting dynamic field information, the target data table including dynamic fields defined by the definition rule, and saving the setting dynamic field information in the metadata of the target data table.

[0008] According to a second aspect of an embodiment of the present invention, a data processing method is provided, comprising: obtaining information about data to be queried requested by a data query request from a data query request, wherein the information about the data to be queried comprises: information about a target data table and information about a target field; obtaining information about a field in the target field that matches set dynamic field information in the metadata based on the information about the target field and metadata of the target data table, or obtaining information about a field that matches the set dynamic field information and information about a fixed field; wherein the set dynamic field information is used to indicate a definition rule for a dynamic field in the target data table; and obtaining a parsing result of parsing the data query request based on the obtained field information.

[0009] According to a third aspect of an embodiment of the present invention, a data processing device is provided, comprising: an instruction receiving module, used to obtain an instruction for creating a target data table, and obtain setting dynamic field information from the creation instruction, wherein the setting dynamic field information is used to indicate a definition rule for a dynamic field in the target data table; a table creation module, used to create the target data table according to the definition rule indicated in the setting dynamic field information, the target data table including dynamic fields defined by the definition rule, and saving the setting dynamic field information into the metadata of the target data table.

[0010] According to a fourth aspect of an embodiment of the present invention, a data processing device is provided, comprising: an acquisition module, used to obtain information about data to be queried requested by the data query request from a data query request, wherein the information about the data to be queried includes: information about a target data table and information about a target field; a matching module, used to obtain information about a field in the target field that matches set dynamic field information in the metadata based on the information about the target field and metadata of the target data table, or to obtain information about a field that matches the set dynamic field information and information about a fixed field; wherein the set dynamic field information is used to indicate a definition rule for a dynamic field in the target data table; and a first generation module, used to obtain a parsing result of parsing the data query request based on the obtained field information.

[0011] According to a fifth aspect of an embodiment of the present invention, there is provided a database system, comprising a front-end node and a computing node; the front-end node is used to obtain a creation instruction of a target data table and obtain set dynamic field information from the creation instruction, the front-end node interacts with the computing node so that the computing node creates the target data table according to the definition rule indicated in the set dynamic field information, the target data table includes dynamic fields defined by the definition rule, and saves the set dynamic field information into metadata of the target data table; or, the front-end node is used to obtain a data query request, and obtain information of the to-be-queried data requested by the data query request and metadata of the target data table to generate a parsing result, the front-end node interacts with the computing node so that the computing node obtains data of the target field requested by the data query request from a data source corresponding to the target data table according to the parsing result.

[0012] According to the sixth aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the data processing method described in the first aspect or the second aspect.

[0013] According to a seventh aspect of an embodiment of the present invention, a computer storage medium is provided, on which at least one target data table is stored, wherein metadata of the target data table includes at least one dynamic field, and the at least one dynamic field is a field created according to a defined rule.

[0014] According to an eighth aspect of an embodiment of the present invention, there is provided a computer storage medium on which a computer program is stored. When the program is executed by a processor, the data processing method as described in the first aspect or the second aspect is implemented.

[0015] According to the data processing solution provided by the embodiment of the present invention, a target data table containing dynamic fields can be created according to the dynamic field information set in the creation instruction, so that the target data table can seamlessly support the changes of the dynamic fields of the data source, and achieve seamless compatibility between data changes and user data query requests, which greatly simplifies user operations and saves performance, so as to solve the problem in the prior art that DDL needs to be used to update the fields in the data table every time the data source changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0017] Figure 1 is a flowchart of a data processing method according to Embodiment 1 of the present invention;

[0018] Figure 2 is a flowchart of a data processing method according to Embodiment 2 of the present invention;

[0019] Figure 3 is a flowchart of a data processing method according to Embodiment 3 of the present invention;

[0020] Figure 4 is a flowchart of a data processing method according to Embodiment 4 of the present invention;

[0021] Figure 5 is a structural block diagram of a data processing device according to a fifth embodiment of the present invention;

[0022] Figure 6 is a structural block diagram of a data processing device according to Embodiment 6 of the present invention;

[0023] Figure 7 is a structural block diagram of a data processing device according to Embodiment 7 of the present invention;

[0024] Figure 8 is a structural block diagram of a data processing device according to Embodiment 8 of the present invention;

[0025] Fig. 9 This is a schematic diagram of a database system in a data lake analysis system scenario according to Embodiment 9 of the present invention;

[0026] Fig.10 This is a schematic diagram of the structure of the first computing node according to Embodiment 9 of the present invention;

[0027] Fig.11 This is a schematic diagram of the structure of the second computing node according to Embodiment 9 of the present invention;

[0028] Fig.12 FIG. 1 is a schematic diagram of the structure of an electronic device according to Embodiment 9 of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0030] The specific implementation of the embodiment of the present invention is further described below in conjunction with the accompanying drawings of the embodiment of the present invention.

[0031] Embodiment 1

[0032] Reference Figure 1 , shows a step flow chart of a data processing method according to embodiment 1 of the present invention.

[0033] The data processing method of this embodiment includes the following steps:

[0034] Step S102: obtaining a creation instruction of a target data table, and obtaining setting dynamic field information from the creation instruction.

[0035] The creation instruction is used to instruct the creation of a target data table in the database. Those skilled in the art can instruct the creation of the target data table in an appropriate manner according to business requirements. For example, SQL is used to instruct the creation of the target data table through the create table keyword.

[0036] The content included in the creation instruction may be different for different instruction methods or different target data table structures. For example, if you need to create a target data table containing dynamic fields, the creation instruction may include the table name of the target data table, set dynamic field information, fixed field information, etc. For another example, if you need to create a target data table containing only fixed fields, the creation instruction may include the table name of the target data table, fixed field information, etc.

[0037] The dynamic field setting information is used to indicate the definition rule of the dynamic field in the target data table. Those skilled in the art can configure appropriate definition rules as needed, and this embodiment does not limit this. For example, the definition rule includes information of a regular expression that defines the dynamic field in the target data table.

[0038] The dynamic field may be a field whose name complies with the definition rules and whose addition or deletion may not change the metadata of the target data table.

[0039] For example, the dynamic field setting information may include a definition rule identifier and dynamic field information. The definition rule identifier is used to indicate the definition rule to which it corresponds, so that the database can create a dynamic field in the target data table according to the definition rule. The dynamic field information includes but is not limited to part or all of the field name of the dynamic field.

[0040] For different creation instructions, the set dynamic field information therein may be obtained in any appropriate manner, for example, by using a lexical analysis method.

[0041] Step S104: creating the target data table according to the definition rule indicated in the set dynamic field information, the target data table including the dynamic fields defined by the definition rule, and saving the set dynamic field information into the metadata of the target data table.

[0042] The definition rule may be preset or obtained from other systems. Taking the case where the definition rule includes information of a regular expression defining a dynamic field in the target data table, the database defines the dynamic field in the target data table (e.g., defines the field name of the dynamic field) according to the definition rule using the regular expression rule and the information of the dynamic field, thereby creating the target data table, and saving the definition rule identifier and the information of the dynamic field in the metadata of the target data table.

[0043] Of course, in other embodiments, those skilled in the art may set any appropriate definition rules according to business needs, and this embodiment does not limit this.

[0044] Optionally, the created target data table can be connected to the data source by using a protocol, path, etc. In this way, the target data table can be loosely coupled with the data source. When the data source changes the fields therein as needed, as long as the changed fields meet the set dynamic field information, the metadata of the target data table can remain unchanged, thereby avoiding the need to change the metadata every time the fields in the data source need to be changed due to the data change in the prior art, thereby reducing the database load.

[0045] Through this embodiment, a target data table containing dynamic fields can be created according to the dynamic field information set in the creation instruction, so that the target data table can seamlessly support changes in the dynamic fields of the data source, achieve seamless compatibility between data changes and user data query requests, greatly simplify user operations, and save performance, so as to solve the problem in the prior art that DDL needs to be used to update the fields in the data table every time the data source changes.

[0046] The data processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as tablet computers, mobile phones, etc.) and PCs, etc.

[0047] Embodiment 2

[0048] Reference Figure 2 , shows a step flow chart of a data processing method according to embodiment 2 of the present invention.

[0049] The data processing method of this embodiment includes the aforementioned steps S102 to S104. In this embodiment, the method further includes:

[0050] Step S106: obtaining information of the to-be-queried data requested by the data query request from the data query request.

[0051] The data query request may be a request generated using a structured query language (such as SQL) for requesting to query and process the data in the database, wherein the processing of the data may be selecting data, deleting data, updating data, and the like.

[0052] The information of the data to be queried can be obtained by performing lexical analysis on the data query request (such as using flex to perform lexical analysis).

[0053] The information of the data to be queried includes: information of the target data table and information of the target field. The target data table may be one or more data tables stored in the database. The information of the target data table may include the table name of the target data table, and may also include other information of the target data table. The information of the target field may include the field names of one or more target fields, and may also include other information of the target field, such as the screening conditions of the target field, etc.

[0054] Step S108: According to the information of the target field and the metadata of the target data table, obtain the information of the field in the target field that matches the set dynamic field information in the metadata, or obtain the information of the field that matches the set dynamic field information and the fixed field information.

[0055] The metadata of the target data table can be understood as the relevant attribute data of the target data table. The metadata includes but is not limited to: the field names of the fields contained in the target data table, the data types of each field, whether it is empty (null), the default value, auto-increment, whether it is the primary key, and comments, etc.

[0056] The dynamic field setting information in the metadata is used to indicate the definition rules of the dynamic fields in the target data table. The dynamic fields can be understood as non-fixed fields defined by corresponding definition rules and can be dynamically increased or decreased in the target data table.

[0057] The dynamic field information may be set in any appropriate manner to indicate the definition rule. For example, in this embodiment, the dynamic field information includes a definition rule identifier for indicating the definition rule and information of the dynamic field for indicating the dynamic field. Of course, the dynamic field information may also include the data type of the dynamic field, whether it is empty, etc., which is not limited in this embodiment.

[0058] The definition rule identifier may be any appropriate identifier as long as it can indicate the corresponding definition rule. For example, the definition rule identifier is "REGEX_COLUMN", which is used to indicate that the definition rule is a regular rule. For another example, the definition rule identifier is "full", which is used to indicate a full-word match definition rule in a preset definition rule, or to indicate a full match definition rule, etc. The definition rule identifier may be set to any appropriate identifier as needed, and this embodiment does not limit this.

[0059] The information of the dynamic field may include a part of the field name of the dynamic field, etc. For example, the information of the dynamic field includes 'abc_.*'. According to the corresponding definition rule, the field name of the dynamic field indicated by the information of the dynamic field starts with 'abc_', such as 'abc_1', 'abc_2', etc.

[0060] By setting dynamic field information in metadata, at least one dynamic field in the target data table is stored. In this way, even if the dynamic field in the target data table changes (such as adding a new field, reducing a new field, or changing the field name), as long as it satisfies the set dynamic field information, the metadata does not need to be modified. This ensures that when the target data table needs to update, add or delete a field, the dynamic field in the target data table can be directly processed accordingly without frequently changing the metadata, thereby reducing the load pressure and resource consumption of the metadata system, and effectively improving the response speed of the database.

[0061] Taking the setting of dynamic field information as REGEX_COLUMN'abc_.*' as an example, the field indicated by the field information in the target field that matches the set dynamic field information in the metadata should be a field whose field name starts with abc, for example, abc_1, abc_2, etc.

[0062] The fixed field information indicates the target field that matches the fixed field of the target data table. For example, the fixed fields of the target data table include name, age, height, etc., and the target field indicated by the fixed field information may be name, age, etc.

[0063] Step S110: Obtain a parsing result of parsing the data query request according to the obtained field information.

[0064] If the target fields indicated by the obtained field information are all the target fields in the target field information, the parsing result of parsing the data query request is directly obtained based on the obtained field information. For example, a parsed structure tree is generated based on the obtained field information.

[0065] If the target field indicated by the obtained field information is part of the target field in the target field information, that is, there is another part of the target field that is neither a fixed field nor a dynamic field in the target data table, then it means that the target field is not in the target data table, and the data query request is incorrect. The parsing can be terminated and an error will be reported.

[0066] Through this embodiment, a target data table containing dynamic fields can be created according to the dynamic field information set in the creation instruction, so that the target data table can seamlessly support changes in the dynamic fields of the data source, achieve seamless compatibility between data changes and user data query requests, greatly simplify user operations, and save performance, so as to solve the problem in the prior art that DDL needs to be used to update the fields in the data table every time the data source changes.

[0067] In addition, for a data query request, information about the requested data to be queried is obtained. Since the set dynamic field information is used to indicate the definition rules of the dynamic fields in the target data table, information about the fields that match the set dynamic field information is obtained by matching the target field with the set dynamic field information, or information about the fields that match the set dynamic field information and information about the fixed fields are obtained, and a parsing result of parsing the data query request is obtained based on the obtained field information, thereby making it possible to complete the parsing of the data query request without frequently changing the metadata, thereby improving ease of use and simplicity.

[0068] The data processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as tablet computers, mobile phones, etc.) and PCs, etc.

[0069] Embodiment 3

[0070] Reference Figure 3 , shows a step flow chart of a data processing method according to embodiment 3 of the present invention.

[0071] The data processing method of this embodiment includes the following steps:

[0072] Step S202: obtaining information of the to-be-queried data requested by the data query request from the data query request.

[0073] The data query request may be a request generated using a structured query language (such as SQL) for requesting to query and process the data in the database, wherein the processing of the data may be selecting data, deleting data, updating data, and the like.

[0074] The information of the data to be queried can be obtained by performing a lexical analysis on the data query request (such as using flex to perform a lexical analysis). The information of the data to be queried includes: information of the target data table and information of the target field. The target data table may be one or more data tables stored in a database. The information of the target data table may include the table name of the target data table, and may also include other information of the target data table. The information of the target field may include the field names of one or more target fields, and may also include other information of the target field, for example, screening conditions of the target field, etc.

[0075] Step S204: According to the information of the target field and the metadata of the target data table, obtain the information of the field in the target field that matches the set dynamic field information in the metadata, or obtain the information of the field that matches the set dynamic field information and the information of the fixed field.

[0076] The metadata of the target data table can be understood as the relevant attribute data of the target data table. The metadata includes but is not limited to: the field names of the fields contained in the target data table, the data types of each field, whether it is empty (null), the default value, auto-increment, whether it is the primary key, and comments, etc.

[0077] The dynamic field setting information in the metadata is used to indicate the definition rules of the dynamic fields in the target data table. The dynamic fields can be understood as non-fixed fields defined by corresponding definition rules and can be dynamically increased or decreased in the target data table.

[0078] The dynamic field information may be set in any appropriate manner to indicate the definition rule. For example, in this embodiment, the dynamic field information includes a definition rule identifier for indicating the definition rule and information of the dynamic field for indicating the dynamic field. Of course, the dynamic field information may also include the data type of the dynamic field, whether it is empty, etc., which is not limited in this embodiment.

[0079] The definition rule identifier may be any appropriate identifier as long as it can indicate the corresponding definition rule. For example, the definition rule identifier is "REGEX_COLUMN", which is used to indicate that the definition rule is a regular rule. For another example, the definition rule identifier is "full", which is used to indicate a full-word match definition rule in a preset definition rule, or to indicate a full match definition rule, etc. The definition rule identifier may be set to any appropriate identifier as needed, and this embodiment does not limit this.

[0080] The information of the dynamic field may include a part of the field name of the dynamic field, etc. For example, the information of the dynamic field includes 'abc_.*'. According to the corresponding definition rule, the field name of the dynamic field indicated by the information of the dynamic field starts with 'abc_', such as 'abc_1', 'abc_2', etc.

[0081] By setting dynamic field information in metadata, at least one dynamic field in the target data table is stored. In this way, even if the dynamic field in the target data table changes (such as adding a new field, reducing a new field, or changing the field name), as long as it satisfies the set dynamic field information, the metadata does not need to be modified. This ensures that when the target data table needs to update, add or delete a field, the dynamic field in the target data table can be directly processed accordingly without frequently changing the metadata, thereby reducing the load pressure and resource consumption of the metadata system, and effectively improving the response speed of the database.

[0082] Taking the setting of dynamic field information as REGEX_COLUMN'abc_.*' as an example, the field indicated by the field information in the target field that matches the set dynamic field information in the metadata should be a field whose field name starts with abc, for example, abc_1, abc_2, etc.

[0083] The fixed field information indicates the target field that matches the fixed field of the target data table. For example, the fixed fields of the target data table include name, age, height, etc., and the target field indicated by the fixed field information may be name, age, etc.

[0084] Step S206: Obtain a parsing result of parsing the data query request according to the obtained field information.

[0085] If the target fields indicated by the obtained field information are all the target fields in the target field information, the parsing result of parsing the data query request is directly obtained based on the obtained field information. For example, a parsed structure tree is generated based on the obtained field information.

[0086] If the target field indicated by the obtained field information is part of the target field in the target field information, that is, there is another part of the target field that is neither a fixed field nor a dynamic field in the target data table, then it means that the target field is not in the target data table, and the data query request is incorrect. The parsing can be terminated and an error will be reported.

[0087] Through this embodiment, in response to a data query request, information about the requested data to be queried is obtained. Since the set dynamic field information is used to indicate the definition rule of the dynamic field in the target data table, information about the field matching the set dynamic field information is obtained by matching the target field with the set dynamic field information, or information about the field matching the set dynamic field information and information about the fixed field are obtained, and a parsing result of parsing the data query request is obtained based on the obtained field information, so that the parsing of the data query request can be completed without frequently changing the metadata, thereby improving ease of use and simplicity.

[0088] The data processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as tablet computers, mobile phones, etc.) and PCs, etc.

[0089] Embodiment 4

[0090] Reference Figure 4 , shows a step flow chart of a data processing method according to embodiment 4 of the present invention.

[0091] The data processing method of this embodiment includes the following steps:

[0092] Step S300a: Obtain the creation instruction of the target data table, and obtain the set dynamic field information from the creation instruction.

[0093] It should be noted that this step is an optional step, which can be performed at any appropriate time before the following step S302.

[0094] The creation instruction is used to instruct the creation of a target data table in the database. Those skilled in the art can instruct the creation of the target data table in an appropriate manner according to business requirements. For example, SQL is used to instruct the creation of the target data table through the create table keyword.

[0095] The content included in the creation instruction may be different for different instruction methods or different target data table structures. For example, if you need to create a target data table containing dynamic fields, the creation instruction may include the table name of the target data table, set dynamic field information, fixed field information, etc. For another example, if you need to create a target data table containing only fixed fields, the creation instruction may include the table name of the target data table, fixed field information, etc.

[0096] The dynamic field setting information includes a definition rule identifier and dynamic field information. The definition rule identifier is used to indicate the definition rule to which it corresponds, so that the database can create a dynamic field in the target data table according to the definition rule. The dynamic field information includes but is not limited to part or all of the field name of the dynamic field.

[0097] For different creation instructions, the set dynamic field information therein may be obtained in any appropriate manner, for example, by using a lexical analysis method.

[0098] Step S300b: creating the target data table according to the definition rule identifier and the information of the dynamic field in the information for setting the dynamic field, and saving the definition rule identifier and the information of the dynamic field into the metadata of the target data table.

[0099] It should be noted that this step is an optional step, which can be performed at any appropriate time after step S300a and before step S302 described below.

[0100] The database may determine the corresponding definition rule according to the definition rule identifier. The definition rule may be a preset definition rule or may be obtained from other systems according to the definition rule identifier.

[0101] Taking the example of a definition rule including information of a regular expression defining a dynamic field in the target data table, the database defines the dynamic field in the target data table (e.g., defines the field name of the dynamic field) using the regular expression rule and the information of the dynamic field according to the definition rule, thereby creating the target data table and saving the definition rule identifier and the information of the dynamic field to the metadata of the target data table.

[0102] Of course, in other embodiments, those skilled in the art may set any appropriate definition rules according to business needs, and this embodiment does not limit this.

[0103] Optionally, the created target data table can be connected to the data source by using a protocol, path, etc. In this way, the target data table can be loosely coupled with the data source. When the data source changes the fields therein as needed, as long as the changed fields meet the set dynamic field information, the metadata of the target data table can remain unchanged, thereby avoiding the need to change the metadata every time the fields in the data source need to be changed due to the data change in the prior art, thereby reducing the database load.

[0104] Step S302: obtaining information of the to-be-queried data requested by the data query request from the data query request.

[0105] This step can be implemented in the same manner as step S202 in the third embodiment, so it will not be described in detail here.

[0106] Step S304: According to the information of the target field and the metadata of the target data table, obtain the information of the field in the target field that matches the set dynamic field information in the metadata, or obtain the information of the field that matches the set dynamic field information and the fixed field information.

[0107] In a first specific implementation, step S304 includes the following sub-steps:

[0108] Sub-step S3041: According to the information of the target field, determine the fields in the target field that do not match the fixed fields in the target data table.

[0109] In a specific embodiment, based on the information of the target field, the indicated target field is matched with the fixed field in the target data table. For example, the field name of the target field is matched with the field name of the fixed field in the target data table using an exact match method, and the target field whose field name does not match the field name of the fixed field is determined as the field in the target field that does not match the fixed field in the target data table.

[0110] Sub-step S3042: Match the information of the unmatched field with the set dynamic field information, and obtain the information of the field that matches the set dynamic field information according to the matching result.

[0111] For the information of the unmatched field, match it with the set dynamic field information. For example, according to the definition rule indicated by the set dynamic field information, determine the matching method between the information of the unmatched field and the set dynamic field information, and match the field name of the target field indicated by the information of the unmatched field with the information of the dynamic field indicated by the set dynamic field information. According to the matching result, obtain the information of the field that matches the set dynamic field information.

[0112] For example, if the definition rule includes the information of the regular expression defining the dynamic field in the target data table, the corresponding matching method is the regular matching method. If the dynamic field information includes 'abc_.*', then the target fields whose field names begin with 'abc_' can be determined as the target fields that match the set dynamic field information, and then the matching result can be determined accordingly, and the information of the field that matches the set dynamic field information can be obtained according to the matching result.

[0113] It should be noted that the metadata may include multiple set dynamic field information. When matching, the information of the unmatched fields may be matched one by one with the set dynamic field information.

[0114] In this way, the information of the field in the target field that matches the set dynamic field information in the metadata can be quickly and accurately obtained, thereby improving the matching efficiency. Moreover, the information of different set dynamic fields can be matched using the corresponding matching methods, thereby ensuring the accuracy of the matching.

[0115] Alternatively, in the second specific implementation, the step S304, in addition to the sub-steps S3041 to S3042 in the first specific implementation, further includes:

[0116] Sub-step S3043: Determine the information of the fixed field in the target field.

[0117] For example, based on the information of the target field, the target field indicated by it is matched with the fixed field in the target data table. For example, using an exact matching method, the field name of the target field is matched with the field name of the fixed field in the target data table, and the target field whose field name matches the field name of the fixed field is determined as the fixed field in the target field, and the information of the fixed field is obtained.

[0118] It should be noted that, in order to improve efficiency, the matching result of sub-step S3041 can be directly used in sub-step S3043 to determine the target field whose field name matches the field name of the fixed field as the fixed field in the target field, and obtain the information of the fixed field.

[0119] Step S306: Obtain a parsing result of parsing the data query request according to the obtained field information.

[0120] This step can be implemented in the same manner as step S206 in the third embodiment, so it will not be described in detail here.

[0121] Step S308: Generate a query execution instruction corresponding to the data query request according to the analysis result.

[0122] It should be noted that this step is an optional step, which can be performed at any appropriate time after step S306.

[0123] The query execution instruction is used to instruct to perform a corresponding query action, such as scanning the data source of the target data table, etc. Those skilled in the art may generate the query instruction according to the parsing result in any appropriate manner.

[0124] Optionally, in order to improve the response speed to data query requests, the database generates an execution plan with minimal resource consumption based on the analysis results. The execution plan includes one or more query execution instructions. Compared with the method of directly generating query execution instructions, this method can optimize the order of query execution instructions, thereby further reducing the resource consumption of executing query execution instructions.

[0125] Step S310: According to the query execution instruction, the data of the target field requested by the data query request is obtained from the data source corresponding to the target data table.

[0126] In a specific implementation, the step S310 includes the following sub-steps:

[0127] Sub-step S3101: According to the query execution instruction, determine whether the information of the target field contains a wildcard identifier for indicating all fields.

[0128] Taking SQL as an example, the wildcard identifier for indicating all fields may be '*'. Of course, those skilled in the art may also define other characters as the wildcard identifier for indicating all fields.

[0129] If it is determined that the information of the target field contains a wildcard identifier, sub-step S3102 is executed; if it is determined that the information of the target field does not contain a wildcard identifier, the data of the target field involved in the query execution instruction is directly obtained from the data source corresponding to the target data table.

[0130] Sub-step S3102: If included, obtain information of all fields in the data source corresponding to the target data table.

[0131] Since the metadata of the target data table stores the dynamic fields in the target data table by setting dynamic field information, it is impossible to directly determine which dynamic fields are indicated by the wildcard identifier based on the metadata. Therefore, when a wildcard identifier is included, information on all fields in the data source corresponding to the target data table is obtained to determine the field indicated by the wildcard identifier based on information on all fields actually existing in the data source of the target data table.

[0132] Sub-step S3103: Match the information of all the fields with the set dynamic field information, and determine the dynamic field in all the fields according to the matching result.

[0133] In a specific implementation, when the information of all fields matches the set dynamic field information, a matching method corresponding to the definition rule indicated by the set dynamic field information can be used to match the field names in the information of all fields with the dynamic field information, and all matching fields are determined as dynamic fields in all fields.

[0134] For example, if the matching method is regular matching and the dynamic field information is 'abc_.*', then all fields whose field names begin with 'abc_' can be determined as dynamic fields. In this way, the wildcard identifier can be expanded to determine the specific field it indicates.

[0135] Sub-step S3104: acquiring data of all fields including the dynamic field from the data source corresponding to the target data table.

[0136] According to the field indicated by the expanded wildcard identifier, data of all fields including the dynamic field are obtained from the data source corresponding to the target data table.

[0137] In this way, for the target data table containing dynamic fields, if the query execution instruction involves a wildcard identifier, the dynamic field method can be determined by scanning the fields in the data source and matching these fields with the set dynamic field information, and then the set dynamic field information in the metadata can be expanded to determine the specific field corresponding to the wildcard identifier, and then all the fields corresponding to the wildcard identifier can be obtained from the data source.

[0138] Through this embodiment, in response to a data query request, information about the requested data to be queried is obtained. Since the set dynamic field information is used to indicate the definition rule of the dynamic field in the target data table, information about the field matching the set dynamic field information is obtained by matching the target field with the set dynamic field information, or information about the field matching the set dynamic field information and information about the fixed field are obtained, and a parsing result of parsing the data query request is obtained based on the obtained field information, so that the parsing of the data query request can be completed without frequently changing the metadata, thereby improving ease of use and convenience.

[0139] Embodiment 5

[0140] Reference Figure 5 , shows a structural block diagram of a data processing device according to Embodiment 5 of the present invention.

[0141] The data processing device of this embodiment includes: an instruction receiving module 502, which is used to obtain a creation instruction of a target data table, and obtain set dynamic field information from the creation instruction, wherein the set dynamic field information is used to indicate a definition rule of a dynamic field in the target data table; a table creation module 504, which is used to create the target data table according to the definition rule indicated in the set dynamic field information, the target data table includes the dynamic fields defined by the definition rule, and saves the set dynamic field information into the metadata of the target data table.

[0142] Through this embodiment, a target data table containing dynamic fields can be created according to the dynamic field information set in the creation instruction, so that the target data table can seamlessly support changes in the dynamic fields of the data source, achieve seamless compatibility between data changes and user data query requests, greatly simplify user operations, and save performance, so as to solve the problem in the prior art that DDL needs to be used to update the fields in the data table every time the data source changes.

[0143] Embodiment 6

[0144] Reference Figure 6 , shows a structural block diagram of a data processing device according to embodiment 6 of the present invention.

[0145] The data processing device of this embodiment includes: an instruction receiving module 602, which is used to obtain a creation instruction of a target data table, and obtain set dynamic field information from the creation instruction, wherein the set dynamic field information is used to indicate a definition rule of a dynamic field in the target data table; a table creation module 604, which is used to create the target data table according to the definition rule indicated in the set dynamic field information, the target data table includes the dynamic fields defined by the definition rule, and saves the set dynamic field information into the metadata of the target data table.

[0146] Optionally, the definition rule includes information of a regular expression defining a dynamic field in the target data table.

[0147] Optionally, the device also includes: an acquisition module 606, used to obtain information about the data to be queried requested by the data query request from the data query request, wherein the information about the data to be queried includes: information about the target data table and information about the target field; a matching module 608, used to obtain information about the field in the target field that matches the set dynamic field information in the metadata based on the information about the target field and the metadata of the target data table, or to obtain information about the field that matches the set dynamic field information and information about the fixed field; a first generation module 610, used to obtain a parsing result of parsing the data query request based on the obtained field information.

[0148] The data processing device of this embodiment is used to implement the corresponding data processing methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0149] Embodiment 7

[0150] Reference Figure 7 , shows a structural block diagram of a data processing device according to Embodiment 7 of the present invention.

[0151] The data processing device of this embodiment includes: an acquisition module 702, which is used to obtain information about the data to be queried requested by the data query request from the data query request, wherein the information about the data to be queried includes: information about the target data table and information about the target field; a matching module 704, which is used to obtain information about the field in the target field that matches the set dynamic field information in the metadata based on the information about the target field and the metadata of the target data table, or to obtain information about the field that matches the set dynamic field information and information about the fixed field; wherein the set dynamic field information is used to indicate the definition rule of the dynamic field in the target data table; a first generation module 706, which is used to obtain a parsing result of parsing the data query request based on the obtained field information.

[0152] Through this embodiment, in response to a data query request, information about the requested data to be queried is obtained. Since the set dynamic field information is used to indicate the definition rule of the dynamic field in the target data table, information about the field matching the set dynamic field information is obtained by matching the target field with the set dynamic field information, or information about the field matching the set dynamic field information and information about the fixed field are obtained, and a parsing result of parsing the data query request is obtained based on the obtained field information, so that the parsing of the data query request can be completed without frequently changing the metadata, thereby improving ease of use and simplicity.

[0153] Embodiment 8

[0154] Reference Figure 8 , shows a structural block diagram of a data processing device according to embodiment 8 of the present invention.

[0155] The data processing device of this embodiment includes: an acquisition module 802, which is used to obtain information about the to-be-queried data requested by the data query request from the data query request, wherein the information about the to-be-queried data includes: information about the target data table and information about the target field; a matching module 804, which is used to obtain information about the field in the target field that matches the set dynamic field information in the metadata based on the information about the target field and the metadata of the target data table, or to obtain information about the field that matches the set dynamic field information and information about the fixed field; wherein the set dynamic field information is used to indicate the definition rule of the dynamic field in the target data table; a first generation module 806, which is used to obtain a parsing result of parsing the data query request based on the obtained field information.

[0156] Optionally, the definition rule includes information of a regular expression defining a dynamic field in the target data table.

[0157] Optionally, the matching module 804 is used to determine, based on the information of the target field, the fields in the target field that do not match the fixed fields in the target data table; match the information of the unmatched fields with the set dynamic field information, and obtain information of the fields that match the set dynamic field information based on the matching result; or, based on the information of the target field and the metadata of the target data table, obtain information of the fields that match the set dynamic field information and information of the fixed fields; wherein the set dynamic field information is used to indicate the definition rules of the dynamic fields in the target data table.

[0158] Optionally, the device also includes: a second generation module 808, used to generate a query execution instruction corresponding to the data query request based on the analysis result; a data acquisition module 810, used to obtain the data of the target field requested by the data query request from the data source corresponding to the target data table according to the query execution instruction.

[0159] Optionally, the data acquisition module 810 includes: a wildcard determination module 8101, used to determine whether the information of the target field contains a wildcard identifier for indicating all fields according to the query execution instruction; a field information acquisition module 8102, used to obtain the information of all fields in the data source corresponding to the target data table when it is contained; a dynamic field determination module 8103, used to match the information of all fields with the set dynamic field information, and determine the dynamic fields in all fields according to the matching results; a field data extraction module 8104, used to obtain the data of all fields including the dynamic fields from the data source corresponding to the target data table.

[0160] The data processing device of this embodiment is used to implement the corresponding data processing methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0161] Embodiment 9

[0162] See also Fig. 9 , which shows a schematic diagram of a database system used in a data lake analysis system (Data Lake Analytics) scenario. The database system includes a front node (FRONTNODE) ​​and a computing node (COMPUTENODE).

[0163] The front node is used to obtain a creation instruction for a target data table and obtain setting dynamic field information from the creation instruction. The front node interacts with the computing node so that the computing node creates the target data table according to the definition rule indicated in the setting dynamic field information, the target data table includes the dynamic fields defined by the definition rule, and saves the setting dynamic field information into the metadata of the target data table.

[0164] In this way, a target data table containing dynamic fields can be created according to the dynamic field information set in the creation instruction, so that the target data table can seamlessly support changes in the dynamic fields of the data source, achieve seamless compatibility between data changes and user data query requests, greatly simplify user operations, and save performance, so as to solve the problem in the prior art that DDL needs to be used to update the fields in the data table every time the data source changes.

[0165] Alternatively, the front node is used to obtain a data query request, and obtain information about the data to be queried requested by the data query request and metadata of the target data table to generate a parsing result. The front node interacts with the computing node so that the computing node obtains the data of the target field requested by the data query request from the data source corresponding to the target data table according to the parsing result.

[0166] In this way, for a data query request, information about the requested data to be queried is obtained. Since the set dynamic field information is used to indicate the definition rules of the dynamic fields in the target data table, information about the fields matching the set dynamic field information is obtained by matching the target field with the set dynamic field information, or information about the fields matching the set dynamic field information and information about the fixed fields are obtained, and a parsing result of the data query request is obtained based on the obtained field information, so that the parsing of the data query request can be completed without frequently changing the metadata, thereby improving ease of use and simplicity.

[0167] The front node can be used to execute step S102, step S106 to step S110 in the above method embodiment and achieve corresponding effects. The computing node can be used to execute step S104 in the above method embodiment and achieve corresponding effects.

[0168] Alternatively, the front node can also be used to execute steps S202 to S206, step S300a, and steps S302 to S308 in the above method embodiment and achieve corresponding effects. The computing node can be used to execute steps S300b and S310 in the above method embodiment and achieve corresponding effects.

[0169] In a specific implementation, the database system includes multiple front-end nodes and multiple computing nodes. The multiple front-end nodes are mounted on a load balancing device (SLB) to achieve load balancing of request processing.

[0170] Multiple front nodes serve each other to form a front node pool (FRONTNODE Pool). The front node receives data query requests (Query), parses them, generates and optimizes distributed execution plans, etc. After the distributed execution plan is generated, it is distributed and scheduled to multiple computing nodes for execution. Multiple computing nodes form a computing node pool (COMPUTENODEPool). The entire database system architecture adopts a storage and computing separation architecture, that is, the data source (Data Source) is separated from the computing service cluster of the database system (the computing service cluster at least includes the computing node pool), and the data table in the database system uses dynamic fields created by defining rules to associate one or more fields in the data source, so that the fields in the data source can be changed independently without frequently changing the fields of the data table in the database system. At runtime (runtime), the computing node reads data from the data source and binds the fields in the data source corresponding to the dynamic fields according to the matching results between the dynamic fields and the fields in the data source.

[0171] In this embodiment, the database system executes the aforementioned method to create a target data table containing dynamic fields, for example:

[0172] Step 1: Get and set dynamic field information according to the creation instruction.

[0173] Taking SQL as an example, the creation instruction is:

[0174] create table users(

[0175] name varchar(120),

[0176] age varchar(20),

[0177] regex_column 'abc_.*',

[0178] regex_column'xy[1-3]? ', )

[0180] The keyword create table is used to indicate the creation of a table. users is the table name of the target data table. name and age are the field names of fixed fields. varchar indicates that the data type of the corresponding field is a variable-length string type. regex_column is the definition rule identifier in the dynamic field information setting. 'abc_.*' and 'xy[1-3]? ' are the information of the dynamic fields in the dynamic field information setting. regex_column'abc_.*' refers to all fields starting with abc_. regex_column'xy[1-3]? ' refers to the fields with field names "xy, xy1, xy2, xy3".

[0181] Step 2: Create the target data table according to the definition rule identifier and the dynamic field information in the information for setting the dynamic field, and save the definition rule identifier and the dynamic field information into the metadata of the target data table.

[0182] Fixed fields in the target data table are created according to the creation instruction, and dynamic fields in the target data table are created according to the definition rule identifier and the information of the dynamic field in the information for setting the dynamic field, so as to create the target data table, and associate each field of the target data table with the data source.

[0183] It should be noted that, depending on the creation instruction, the target data table may contain only dynamic fields or only fixed fields.

[0184] The definition rule identifier and the dynamic field information are saved in the metadata of the target data table. The fixed field information is saved in the metadata of the target data table. Fig. 9 In the metadata module in . For example, metadata includes:

[0185] name varchar(120),

[0186] age varchar(20),

[0187] regex_column 'abc_.*',

[0188] regex_column'xy[1-3]? ',wait.

[0189] It should be noted that if the data type of a dynamic field is uncertain or inconsistent, its data type can be configured as a binary type or a string type in the metadata.

[0190] In this embodiment, the database system executes the above method, and the process of performing data query on the created target data table is as follows:

[0191] Step A: The load balancing device receives the data query request and distributes the data query request to the front-end node in the front-end node pool, and the front-end node parses the data query request and obtains the parsing result.

[0192] The load balancing device performs load balancing based on the resources of each front-end node and distributes data query requests to the appropriate front-end node.

[0193] The process of the front-end node parsing the data query request is as follows:

[0194] Step A1: obtaining information of the to-be-queried data requested by the data query request from the data query request.

[0195] For example, a lexical analysis is performed on the data query request to obtain information about the data to be queried requested by the data query request. The information about the data to be queried includes information about the target data table and information about the target field.

[0196] For example, the data query request is: select name, abc_3 from users where age>10 and xy1in('abc','zzz').

[0197] The target data table information obtained includes the table name, i.e., users. The target field information includes the field names of the target fields, i.e., name, abc_3, age, and xy1.

[0198] Step A2: Based on the information of the target field and the metadata of the target data table, obtain the information of the field in the target field that matches the set dynamic field information in the metadata, or obtain the information of the field that matches the set dynamic field information and the fixed field information.

[0199] When obtaining the field information in the target field that matches the set dynamic field information in the metadata based on the information of the target field and the metadata of the target data table, first match the field name of the target field with the field name of the fixed field in the metadata. If matching name with the field name of the fixed field is successful, the information of the fixed field can be obtained based on the information of the successfully matched target field.

[0200] Matching abc_3 and xy1 with the field names of the fixed fields will fail to match successfully, and these target fields that cannot be matched successfully are determined as unmatched fields.

[0201] The information of these unmatched fields is matched with the set dynamic field information, and the information of the fields matching the set dynamic field information is obtained according to the matching result. That is, abc_3 and xy1 are matched with regex_column'abc_.*' and regex_column'xy[1-3]? respectively. If the matching is successful, the information of the fields matching the set dynamic field information is obtained according to the information of the two target fields.

[0202] During the parsing process, target fields that cannot be matched with fixed fields are matched with set dynamic field information according to rules, and the set dynamic field information can indicate multiple dynamic fields defined with the same definition rules. Therefore, even if a field in the data source is temporarily deleted or added, the metadata does not need to be changed, and correct matching is still guaranteed. Therefore, the metadata does not need to be changed when the field in the data source is changed, which reduces the load and ensures the correct processing of data query requests.

[0203] For example, if the data source contains field names abc_1, abc_2, and abc_3, when the abc_3 field in the data source is deleted, there is no need to change the metadata.

[0204] Step A3: Obtain a parsing result of parsing the data query request according to the obtained field information.

[0205] The parsing result is generated and obtained according to the obtained field information and the keywords in the data query request. In this usage scenario, the parsing result can be a structure tree.

[0206] Step B: Analyze and process the parsing results.

[0207] According to the parsing result, it is analyzed and processed. For example, an index of the target field involved in the parsing result is established, an index of the target data table involved in the parsing result is established, etc., so as to facilitate subsequent processing. Of course, those skilled in the art can omit this step as needed.

[0208] Step C: Rewrite the data query request according to the parsing result.

[0209] The rewrite operation is used to rewrite the data query request into a more efficient query instruction. For example, a rewriter can be used to perform an equivalent transformation on the data query request so that the transformed query instruction is more efficient and consumes less resources.

[0210] Step D: Generate an execution plan.

[0211] According to the rewritten query instructions and metadata, the non-emptiness, uniqueness, order, and type of the fields involved in the query instructions are determined. For example, if the query instruction contains a clause to query the null value in a certain field, but when performing the non-emptiness judgment, it is determined that the attributes of the field stipulate that it is a non-empty column, the execution result of the clause can be directly determined as a null value without actually executing the clause, which helps improve efficiency.

[0212] Afterwards, an execution plan is generated according to the judgment result and the physical resources, and the execution plan includes one or more query execution instructions. Each query execution instruction can be used to instruct the execution of a query clause.

[0213] Step E: Distribute and schedule the execution plan to the corresponding computing nodes.

[0214] Scheduling is performed according to the generated execution plan, and the query execution instructions therein are distributed to the corresponding computing nodes.

[0215] Step F1: The computing node executes the execution plan.

[0216] The schematic diagram of the structure of the computing node used to execute the execution plan without wildcard identifier is as follows Fig.10 For example, the data query request is select name, abc_3 from users where age>10and xy1 in('abc','zzz').

[0217] The table scan operator of the computing node obtains data of fields named age, name, xy1, and abc_3 from the target data table (recorded as data set A1). The filter operator obtains data of rows with age>10 from data set A1 (recorded as data set B1), and obtains data of rows containing 'abc', 'zzz' in the xy1 field based on data set B1 (recorded as data set C1). The engineer obtains data of fields named name and abc_3 from data set C1 (recorded as data set D1), and the outputter outputs data set D1 as the execution result.

[0218] Step G: The result manager displays the result according to the execution result.

[0219] The result manager can process the execution results as needed and generate the final display results, for example, adjusting the display layout.

[0220] The query processing process of the data query request containing a wildcard identifier is similar to the processing process of the data query request not containing a wildcard identifier, except that step H2 is used instead of step H1.

[0221] Step F2: The schematic diagram of the structure of the computing node used to execute the execution plan with the wildcard identifier is as follows Fig.11 For example, the data query request is select * from users where age>10.

[0222] The scanner (table scan operator) of the computing node obtains information of all fields from the target data table, such as the field names of the obtained fields: name, age, abc_1, abc_2, abc_3, xy1, etc. The information of all fields is matched with the set dynamic field information, and the dynamic fields in all fields are determined according to the matching results. Since abc_1, abc_2, abc_3 can match regex_column'abc_.*', and xy1 can match regex_column'xy[1-3]? ', the dynamic fields include fields with field names of abc_1, abc_2, abc_3, and xy1. * is expanded according to the dynamic fields, and it is determined that all fields indicated by * include field names of name, age, abc_1, abc_2, abc_3, xy1, etc. The data of all fields including the dynamic fields are obtained. That is, obtain the data of the fields named age, name, xy1, abc_1, abc_2, and abc_3 (recorded as data set A2).

[0223] The filter operator obtains the data of the rows with age>10 from the data set A2 (referred to as data set B2).

[0224] Since there is no need to filter the fields, the projector can omit this or not act. The outputter outputs the data set B2 as the execution result.

[0225] In this embodiment, the execution plan for a single data query request will scan the data in the data source and pass the data upward; if the fields designed in the execution plan are certain, they can be pushed down to the underlying operators at the beginning of the calculation, for example, the scanner can filter the fields when pulling data at the bottom layer. If the fields involved are uncertain (that is, when wildcard identifiers are included), then this information is always passed down, and the scanner pulls all the row data in the target data table from the bottom layer, and then reversely determines the specific fields involved, so that the target data table containing dynamic fields can be queried normally and reliably.

[0226] In this embodiment, by creating a target data table with dynamic fields, the target data table does not need to clearly define all fields like a traditional database data table. In the data lake scenario, the target data table with dynamic fields allows users to define an uncertain table structure and define a series of dynamic field information (i.e., field rules). These field rules can participate in the calculation when parsing, optimizing, and executing data query requests, and make specific comparisons and judgments.

[0227] For example, when parsing a data query request, it is necessary to determine whether the field involved in the data query request exists in the target data table. In this case, the field can be matched with the fixed field of the target data table in the metadata. If a match is found, it is determined that the field exists in the target data table. Otherwise, a match is performed on the dynamic field rules set in the metadata. If a match is found, it is considered that the field exists in the target data table. If no match is found, it is determined that the field does not exist in the target data table, and an error can be directly reported.

[0228] In this embodiment, if a field involved in a data query request matches the set dynamic field information, but the data source does not include a field matching the set dynamic field information, the value of the field is determined to be empty.

[0229] For ambiguous data query requests, an error is reported during parsing. For example, the data query request is selectabc_1 from users1 join users2, where users1 and users2 are both target data tables with dynamic field definitions, and both users1 and users2 have abc_1 fields. In this case, the data query request can be rewritten as: selectuser1.abc_1 from users1 join users2.

[0230] For the case where the data type of the field involved in the data query request is uncertain, whether the field appears in a function, expression, or selection field, the type can be inferred and converted based on the current context. The content of any field can be predetermined as binary or string type, and then type conversion can be performed based on the requirements. If the conversion fails, the value of the current expression is null.

[0231] By supporting dynamic field definition, it is possible to minimize repeated changes in table structure, control the width of table structure, etc., and seamlessly support the docking needs of a large number of wide tables and data sources with many fields in data lake scenarios; within the entire computing engine, it supports dynamic field recognition, definition rule judgment, underlying data reverse deduction and other logic, thereby realizing seamless use of dynamic fields, greatly facilitating the application of data lake users.

[0232] Embodiment 10

[0233] Reference Fig.12 , shows a schematic diagram of the structure of an electronic device according to Embodiment 10 of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

[0234] like Fig.12 As shown, the electronic device may include: a processor (processor) 1202 , a communication interface (Communications Interface) 1204 , a memory (memory) 1206 , and a communication bus 1208 .

[0235] in:

[0236] The processor 1202 , the communication interface 1204 , and the memory 1206 communicate with each other via a communication bus 1208 .

[0237] The communication interface 1204 is used to communicate with other electronic devices such as terminal devices or servers.

[0238] The processor 1202 is used to execute the program 1210, and specifically can execute the relevant steps in the above data processing method embodiment.

[0239] Specifically, the program 1210 may include program codes, which include computer operation instructions.

[0240] The processor 1202 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0241] The memory 1206 is used to store the program 1210. The memory 1206 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0242] Program 1210 can be specifically used to enable processor 1202 to perform the following operations: obtain a creation instruction for a target data table, obtain setting dynamic field information from the creation instruction, wherein the setting dynamic field information is used to indicate a definition rule for a dynamic field in the target data table; create the target data table according to the definition rule indicated in the setting dynamic field information, the target data table including the dynamic fields defined by the definition rule, and save the setting dynamic field information in the metadata of the target data table.

[0243] In a feasible manner, the definition rule includes information of a regular expression defining the dynamic field in the target data table.

[0244] In one feasible embodiment, program 1210 can also be used to enable processor 1202 to obtain information about the data to be queried requested by the data query request from the data query request, wherein the information about the data to be queried includes: information about the target data table and information about the target field; based on the information about the target field and the metadata of the target data table, obtain information about the field in the target field that matches the set dynamic field information in the metadata, or obtain information about the field that matches the set dynamic field information and information about the fixed field; based on the obtained field information, obtain a parsing result of parsing the data query request.

[0245] or,

[0246] Program 1210 can be specifically used to enable processor 1202 to perform the following operations: obtain information about the data to be queried requested by the data query request from a data query request, wherein the information about the data to be queried includes: information about a target data table and information about a target field; obtain information about a field in the target field that matches the set dynamic field information in the metadata based on the information about the target field and metadata of the target data table, or obtain information about a field that matches the set dynamic field information and information about a fixed field; wherein the set dynamic field information is used to indicate a definition rule for a dynamic field in the target data table; and obtain a parsing result of parsing the data query request based on the obtained field information.

[0247] In a feasible manner, the definition rule includes information of a regular expression defining the dynamic field in the target data table.

[0248] In one feasible manner, program 1210 can also be used to enable the processor 1202 to obtain information about fields in the target field that match the set dynamic field information in the metadata based on the information of the target field and the metadata of the target data table, and then determine, based on the information of the target field, the fields in the target field that do not match the fixed fields in the target data table; match the information of the unmatched fields with the set dynamic field information, and obtain information about the fields that match the set dynamic field information based on the matching result.

[0249] In one feasible manner, program 1210 can also be used to enable processor 1202 to generate a query execution instruction corresponding to the data query request based on the analysis result; and obtain data of the target field requested by the data query request from the data source corresponding to the target data table according to the query execution instruction.

[0250] In one feasible manner, program 1210 can also be used to enable the processor 1202 to determine whether the information of the target field contains a wildcard identifier for indicating all fields when obtaining the data of the target field requested by the data query request from the data source corresponding to the target data table according to the query execution instruction; if contained, obtain the information of all fields in the data source corresponding to the target data table; match the information of all fields with the set dynamic field information, and determine the dynamic fields among all fields according to the matching results; and obtain the data of all fields including the dynamic fields from the data source corresponding to the target data table.

[0251] Through the electronic device of this embodiment, a target data table containing dynamic fields can be created according to the dynamic field information set in the creation instruction, so that the target data table can seamlessly support changes in the dynamic fields of the data source, and achieve seamless compatibility between data changes and user data query requests, greatly simplifying user operations and saving performance, so as to solve the problem in the prior art that DDL needs to be used to update the fields in the data table every time the data source changes.

[0252] Alternatively, for a data query request, information about the requested data to be queried is obtained. Since the set dynamic field information is used to indicate the definition rules of the dynamic fields in the target data table, information about the fields that match the set dynamic field information is obtained by matching the target field with the set dynamic field information. Alternatively, information about the fields that match the set dynamic field information and information about the fixed fields are obtained, and a parsing result of the data query request is obtained based on the obtained field information, thereby completing the parsing of the data query request without frequently changing the metadata, thereby improving ease of use and convenience.

[0253] In this embodiment, a computer storage medium is provided, on which at least one target data table is stored, wherein metadata of the target data table includes at least one dynamic field, and the at least one dynamic field is a field created according to a definition rule. The computer storage medium may be any appropriate storage medium, such as a CD ROM, a RAM, a floppy disk, a hard disk, or a magneto-optical disk.

[0254] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present invention can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0255] The above-described method according to an embodiment of the present invention may be implemented in hardware, firmware, or as software or computer code that may be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor, or hardware, the data processing method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the data processing method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the data processing method shown herein.

[0256] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present invention.

[0257] The above implementation methods are only used to illustrate the embodiments of the present invention, and are not limitations of the embodiments of the present invention. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present invention. The patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a creation instruction of a target data table, and obtaining setting dynamic field information from the creation instruction, wherein the setting dynamic field information is used to indicate a definition rule of a dynamic field in the target data table; According to the definition rule indicated in the set dynamic field information, create the target data table, the target data table includes the dynamic fields defined by the definition rule, and save the set dynamic field information into the metadata of the target data table; Obtaining information of the data to be queried requested by the data query request from the data query request, wherein the information of the data to be queried includes: information of a target data table and information of a target field; According to the information of the target field and the metadata of the target data table, obtaining the information of the field in the target field that matches the set dynamic field information in the metadata, or obtaining the information of the field that matches the set dynamic field information and the information of the fixed field; Obtaining a parsing result of parsing the data query request according to the obtained field information; According to the information of the target field and the metadata of the target data table, obtaining information of a field in the target field that matches the set dynamic field information in the metadata includes: Determine, according to the information of the target field, the fields in the target field that do not match the fixed fields in the target data table; The information of the unmatched field is matched with the set dynamic field information, and the information of the field matching the set dynamic field information is obtained according to the matching result.

2. The method according to claim 1, characterized in that The definition rule includes information of a regular expression defining a dynamic field in the target data table.

3. A data processing method, characterized in that: include: Obtaining information of the data to be queried requested by the data query request from the data query request, wherein the information of the data to be queried includes: information of a target data table and information of a target field; According to the information of the target field and the metadata of the target data table, obtaining the information of the field in the target field that matches the set dynamic field information in the metadata, or obtaining the information of the field that matches the set dynamic field information and the information of the fixed field; wherein the set dynamic field information is used to indicate the definition rule of the dynamic field in the target data table; Obtaining a parsing result of parsing the data query request according to the obtained field information; According to the information of the target field and the metadata of the target data table, obtaining information of a field in the target field that matches the set dynamic field information in the metadata includes: Determine, according to the information of the target field, the fields in the target field that do not match the fixed fields in the target data table; The information of the unmatched field is matched with the set dynamic field information, and the information of the field matching the set dynamic field information is obtained according to the matching result.

4. The method according to claim 3, characterized in that The definition rule includes information of a regular expression defining a dynamic field in the target data table.

5. The method according to claim 3, characterized in that: The method further comprises: Generating a query execution instruction corresponding to the data query request according to the analysis result; According to the query execution instruction, the data of the target field requested by the data query request is obtained from the data source corresponding to the target data table.

6. The method according to claim 5, characterized in that The step of acquiring the data of the target field requested by the data query request from the data source corresponding to the target data table according to the query execution instruction includes: Determining, according to the query execution instruction, whether the information of the target field contains a wildcard identifier for indicating all fields; If included, obtain information of all fields in the data source corresponding to the target data table; Matching the information of all the fields with the set dynamic field information, and determining the dynamic fields in all the fields according to the matching result; The data of all fields including the dynamic field are obtained from the data source corresponding to the target data table.

7. A data processing device, characterized in that: include: An instruction receiving module, used for obtaining a creation instruction of a target data table, and obtaining setting dynamic field information from the creation instruction, wherein the setting dynamic field information is used for indicating a definition rule of a dynamic field in the target data table; A table creation module, configured to create the target data table according to the definition rule indicated in the set dynamic field information, the target data table including the dynamic fields defined by the definition rule, and save the set dynamic field information into the metadata of the target data table; An acquisition module, used for obtaining information of the to-be-queried data requested by the data query request from the data query request, wherein the information of the to-be-queried data includes: information of a target data table and information of a target field; A matching module, configured to obtain, according to the information of the target field and the metadata of the target data table, the information of the field in the target field that matches the set dynamic field information in the metadata, or to obtain the information of the field that matches the set dynamic field information and the information of the fixed field; A first generating module, configured to obtain a parsing result of parsing the data query request according to the obtained field information; The matching module is specifically used to determine, based on the information of the target field, the fields in the target field that do not match the fixed fields in the target data table; match the information of the unmatched fields with the set dynamic field information, and obtain the information of the fields that match the set dynamic field information based on the matching result.

8. The device according to claim 7, characterized in that The definition rule includes information of a regular expression defining a dynamic field in the target data table.

9. A data processing device, characterized in that: include: An acquisition module, used for obtaining information of the to-be-queried data requested by the data query request from the data query request, wherein the information of the to-be-queried data includes: information of a target data table and information of a target field; a matching module, configured to obtain, according to the information of the target field and the metadata of the target data table, the information of the field in the target field that matches the set dynamic field information in the metadata, or to obtain the information of the field that matches the set dynamic field information and the information of the fixed field; wherein the set dynamic field information is used to indicate the definition rule of the dynamic field in the target data table; A first generating module, for obtaining a parsing result of parsing the data query request according to the obtained field information; The matching module is specifically used to determine, based on the information of the target field, the fields in the target field that do not match the fixed fields in the target data table; match the information of the unmatched fields with the set dynamic field information, and obtain the information of the fields that match the set dynamic field information based on the matching result.

10. The device according to claim 9, characterized in that The definition rule includes information of a regular expression defining a dynamic field in the target data table.

11. The device according to claim 9, characterized in that The device also includes: A second generating module, configured to generate a query execution instruction corresponding to the data query request according to the parsing result; The data acquisition module is used to acquire the data of the target field requested by the data query request from the data source corresponding to the target data table according to the query execution instruction.

12. The device according to claim 11, characterized in that The data acquisition module comprises: A wildcard determination module, configured to determine, according to the query execution instruction, whether the information of the target field contains a wildcard identifier for indicating all fields; A field information acquisition module, used to acquire information of all fields in the data source corresponding to the target data table when included; A dynamic field determination module, used for matching the information of all the fields with the set dynamic field information, and determining the dynamic field in all the fields according to the matching result; The field data extraction module is used to obtain data of all fields including the dynamic field from the data source corresponding to the target data table.

13. A database system, characterized in that: Includes front-end nodes and computing nodes; The front node is used to obtain a creation instruction of a target data table and obtain the set dynamic field information from the creation instruction, the front node interacts with the computing node, so that the computing node creates the target data table according to the definition rule indicated in the set dynamic field information, the target data table includes the dynamic fields defined by the definition rule, and saves the set dynamic field information into the metadata of the target data table; The front node is further used to obtain a data query request, and obtain information of the to-be-queried data requested by the data query request from the data query request, wherein the information of the to-be-queried data includes: information of a target data table and information of a target field; according to the information of the target field and metadata of the target data table, obtain information of a field in the target field that matches the set dynamic field information in the metadata, or obtain information of a field that matches the set dynamic field information and information of a fixed field; according to the obtained field information, obtain a parsing result of parsing the data query request, and the front node interacts with the computing node so that the computing node obtains data of the target field requested by the data query request from a data source corresponding to the target data table according to the parsing result; The front node is specifically used to determine, based on the information of the target field, the fields in the target field that do not match the fixed fields in the target data table; match the information of the unmatched fields with the set dynamic field information, and obtain the information of the fields that match the set dynamic field information based on the matching result.

14. An electronic device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the data processing method as described in any one of claims 1-2, or to perform an operation corresponding to the data processing method as described in any one of claims 3-6.

15. A computer storage medium storing at least one target data table as claimed in any one of claims 1 to 2, wherein metadata of the target data table includes at least one dynamic field, and the at least one dynamic field is a field created according to a definition rule.

16. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the program implements the data processing method as claimed in any one of claims 1 to 2, or, when executed, implements the data processing method as claimed in any one of claims 3 to 6.

Images