Field reading method and device for nested fields

Abstract

The application provides a field reading method and device for nested fields, which comprises the following steps: obtaining first hierarchical structure information from a corresponding original table structure of a computing engine; reconstructing the first hierarchical structure information to generate a reconstructed table structure; obtaining second hierarchical structure information of the reconstructed table structure; obtaining index information of a field to be queried in the second hierarchical structure; and reading the field to be queried in the reconstructed table structure based on the index information. According to the application, the reading of the nested field can be realized without querying useless fields and performing data extraction and conversion operations, thereby improving the field query performance and saving computing resources.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for reading nested fields. Background Technology

[0002] Flink is a real-time computing framework and distributed processing engine for stateful computation on unbounded and bounded data streams. A Flink program takes one or more data inputs, performs logical operations on the data inputs, and finally sends the results to a data receiver. Flink Structured Query Language (SQL) is a development language designed by Flink Real-Time Computing to simplify the computational model and lower the barrier to entry for users of real-time computing, conforming to standard SQL semantics.

[0003] Currently, Flink SQL supports column pruning, which filters out columns that are not needed in operators and is a common optimization strategy in logical plan optimization. While existing technologies can use column pruning to read one or more fields, they cannot prune nested fields when reading Iceberg nested data. For example, consider field A, a nested field with the following type: c4 Row. <su3 Row<ubs Row<um string,uf string,udr string> ,ut string>,su2 Row<utm string,ujm string> Ideally, if the `um` field is needed, it would be read directly. However, currently, all nested fields of the `c4` field are read, and then data extraction is performed before the `um` field value is obtained. Querying useless fields and performing data extraction and transformation operations will reduce field query performance and waste computing resources. Summary of the Invention

[0004] This invention provides a method and apparatus for reading nested fields, which solves the problem that in the prior art, reading nested fields requires querying useless fields and data extraction and transformation operations, which reduces field query performance and wastes computing resources. The invention enables direct reading of the corresponding field to be queried, avoiding reading other redundant fields, thereby improving field query performance and saving computing resources.

[0005] In a first aspect, the present invention provides a method for reading nested fields, including:

[0006] Obtain first-level structure information from the original table structure corresponding to the computing engine; wherein, the first-level structure information includes first field information corresponding to the field to be queried, and a first data structure associated with the first field information; the first data structure is a nested structure;

[0007] The first-level structural information is reconstructed to generate a reconstructed table structure;

[0008] Obtain the second-level structure information of the reconstructed table structure; wherein, the second-level structure information includes second field information corresponding to the field to be queried, and a second data structure associated with the second field information; the second data structure is a nested structure;

[0009] Based on the second field information, obtain the index information of the field to be queried in the second hierarchical structure;

[0010] The field to be queried in the reconstructed table structure is read based on the index information.

[0011] Furthermore, in the method for reading nested fields, obtaining the index information of the field to be queried in the second hierarchical structure based on the second field information further includes:

[0012] Based on the second field information, the field to be queried in the second hierarchical structure information is renamed to determine the second field name of the field to be queried;

[0013] In the second hierarchical structure, determine the index information corresponding to the second field name of each of the fields to be queried.

[0014] Furthermore, in the method for reading nested fields, the step of renaming the field to be queried based on the second hierarchical structure information to determine the second field name of the field to be queried includes:

[0015] Obtain the name of the first field corresponding to each field in the second data structure;

[0016] When the field to be queried is a nested field in the first data structure, the field to be queried is renamed based on the first field name to obtain the second field name of the field to be queried.

[0017] Furthermore, in the method for reading nested fields, determining the index information corresponding to the second field name of each field to be queried in the second hierarchical structure includes:

[0018] Determine each first field name contained in the second field name of the field to be queried, query the position information corresponding to each first field name in the second data structure, and obtain an array composed of multiple position information;

[0019] The array of location information is used as the index information of the field to be queried in the second hierarchical structure.

[0020] Furthermore, in the method for reading nested fields, the step of reconstructing the first-level structure information to generate a reconstructed table structure includes:

[0021] Based on the original table structure where the field to be queried is located, determine the type of the outer field corresponding to the field to be queried;

[0022] When the type of the outer field determines that the field to be queried is a nested field, the field in the first-level structure corresponding to the field to be queried is encapsulated as custom data; when the type of the outer field determines that the field to be queried is not a nested field, the custom data is the field to be queried itself.

[0023] The reconstructed table structure is generated based on the custom data.

[0024] Furthermore, in the method for reading nested fields, the step of generating the reconstructed table structure based on the custom data includes:

[0025] Based on the type of each field in each level of the first hierarchical structure, determine whether each field is a nested type field;

[0026] If a field of the same nested type in a layer has multiple fields, the multiple fields and their corresponding field types in the custom data are merged and stored in the layer where the multiple fields are located, and the reconstructed table structure is obtained.

[0027] Secondly, the present invention also provides a field reading device for nested fields, the device comprising:

[0028] The first acquisition module is used to acquire first-level structure information from the original table structure corresponding to the computing engine; wherein, the first-level structure information includes first field information corresponding to the field to be queried, and a first data structure associated with the first field information; the first data structure is a nested structure.

[0029] The reconstruction module is used to reconstruct the first-level structural information and generate a reconstructed table structure.

[0030] The second acquisition module is used to acquire the second-level structure information of the reconstructed table structure; wherein, the second-level structure information includes second field information corresponding to the field to be queried, and a second data structure associated with the second field information; the second data structure is a nested structure;

[0031] The third acquisition module is used to acquire the index information of the field to be queried in the second hierarchical structure based on the second field information;

[0032] The reading module is used to read the field to be queried in the reconstructed table structure based on the index information.

[0033] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the field reading method for nested fields as described above.

[0034] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the field reading method for nested fields as described above.

[0035] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the field reading method for nested fields as described above.

[0036] The method and apparatus for reading nested fields provided by this invention obtain first-level structure information from the original table structure corresponding to the computing engine; reconstruct the first-level structure information to generate a reconstructed table structure; obtain second-level structure information of the reconstructed table structure; obtain the index information of the field to be queried in the second-level structure; and read the field to be queried in the reconstructed table structure based on the index information. Therefore, this invention eliminates the need to query useless fields and achieves the reading of nested fields by performing data extraction and transformation operations on the reconstructed table structure, thereby improving field query performance and saving computing resources. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1This is one of the flowcharts illustrating the method for reading nested fields provided by this invention;

[0039] Figure 2 This is a schematic diagram of the framework flow of the method for reading nested fields provided by related technologies;

[0040] Figure 3 This is a diagram illustrating the hierarchical structure corresponding to the field to be queried.

[0041] Figure 4 This is a schematic diagram of the framework flow of the method for reading nested fields provided by the present invention;

[0042] Figure 5 This is the second flowchart of the method for reading nested fields provided by the present invention;

[0043] Figure 6 This is a schematic diagram of the structure of the field reading method device for nested fields provided by the present invention;

[0044] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] To facilitate understanding, the terminology involved in this invention will first be explained.

[0047] (1) Flink

[0048] Flink is an open-source computing engine under Apache that can handle streaming tasks, batch tasks, and supports SQL consumption, data processing, and persistence to external storage systems.

[0049] Specifically, Flink is a real-time computing framework and distributed processing engine for stateful computation on unbounded and bounded data streams. A Flink program takes one or more data inputs, performs logical operations on the data inputs, and finally sends the results to a data receiver.

[0050] (2) Flink SQL

[0051] Flink SQL is a development language designed by Flink Real-Time Computing to simplify the computing model and lower the barrier to entry for users to use real-time computing, conforming to standard SQL semantics.

[0052] (3) TableSchema

[0053] TableSchema is a built-in Flink representation of the structure of the current table.

[0054] (4) Iceberg

[0055] Iceberg is an open-source data lake component under Apache.

[0056] (5) Column cropping

[0057] Column pruning is used to filter out columns that are not needed in operators and is a commonly used optimization strategy in logical plan optimization.

[0058] The following is combined with Figure 1 This invention describes a method for reading nested fields.

[0059] Figure 1 This is one of the flowcharts illustrating the method for reading nested fields provided by this invention. It can be understood that... Figure 1 The methods described can be implemented using a field reading device for nested fields. For example... Figure 1 As shown, the method for reading nested fields provided by this invention includes the following steps:

[0060] Step 110: Obtain the first-level structure information from the original table structure corresponding to the computing engine.

[0061] In one embodiment, the computing engine can be the Flink engine.

[0062] The original table structure corresponding to the calculation engine includes hierarchical structure information for the fields to be queried and the hierarchical structure information for the fields not to be queried. For example, if the fields to be queried are c4.su3.ubs.um, c4.su3.ubs.uf, c2.ula, and c4.su2, the original table structure corresponding to the calculation engine includes fields with the following hierarchical structure:

[0063] |--c1:ROW<'ubj'STRING,'uzg'STRING>,

[0064] |--c2:ROW<'uba'STRING, 'ula'STRING, 'uan'STRING>,

[0065] |--c3:ROW<'su1'ROW<'usz'STRING, 'ugz'STRING>, 'su2'ROW<'ucq'STRING, 'usc'STRING>>,

[0066] |--c4:ROW<'su3'ROW<'ubs'ROW<'um'STRING, 'uf'STRING, 'udr'STRING>, 'ut'STRING>, 'su2'ROW<'utm'STRING, 'ujm'STRING>>,

[0067] |--uflag string

[0068] In the above hierarchical structure, the hierarchical structures corresponding to c1, c3, and uflag are all hierarchical structures of fields not to be queried, while the hierarchical structures corresponding to c4 and c2 are hierarchical structures of fields to be queried.

[0069] Wherein, the first hierarchical structure information includes first field information corresponding to the field to be queried, and a first data structure associated with the first field information; the first data structure is a nested structure.

[0070] More specifically, the first field information is used to represent the type and name of each field included in the first hierarchical structure. That is, the first data structure is used to represent the hierarchy of each field included in the first hierarchical structure, and the type and name of the fields contained in each hierarchy. For ease of understanding, the first hierarchical structure of the fields and the first data structure are illustrated below.

[0071] Based on the example above, the field to be queried is c4.su3.ubs.um, and the first-level structure corresponding to the field c4.su3.ubs.um is:

[0072] c4:ROW<'su3'ROW<'ubs'ROW<'um'STRING, 'uf'STRING, 'udr'STRING>, 'ut'STRING>, 'su2'ROW<'utm'STRING, 'ujm'STRING>>,

[0073] In this structure, ROW and STRING represent the field types, while c4, su3, ubs, um, uf, udr, ut, su2, utm, and ujm are the field names. The corresponding first data structure is as follows: under the ROW type c4 field, there are ROW type su3 and su2 fields; under the su3 field, there are ROW type ubs and STRING type ut fields, where the ubs field includes STRING type um, uf, and udr fields; under the su2 field, there are STRING type utm and ujm fields. It can be seen that the first data structure is a nested structure.

[0074] Step 120: Reconstruct the first-level structure information to generate a reconstructed table structure.

[0075] It is understandable that the restructured table structure only includes the hierarchical structure of the fields to be queried. There can be one or more fields to be queried. When there is only one field to be queried, there is one corresponding first-level structure and one first data structure; when there are multiple fields to be queried, there is one or more corresponding first-level structures and one first data structure.

[0076] In one embodiment, when the fields to be queried are c4.su3.ubs.um, c4.su3.ubs.uf, and c4.su2, the first hierarchical structure and the first data structure corresponding to the fields to be queried, c4.su3.ubs.um, c4.su3.ubs.uf, and c4.su2, can all be the same as the first hierarchical structure and the first data structure corresponding to c4.su3.ubs.um, which will not be elaborated here.

[0077] In another embodiment, when the fields to be queried are c4.su3.ubs.um, c4.su3.ubs.uf, c2.ula, and c4.su2, the first-level structure and first data structure corresponding to the fields c4.su3.ubs.um, c4.su3.ubs.uf, and c4.su2 can be the same as the first-level structure and first data structure corresponding to c4.su3.ubs.um, which will not be repeated here. The first-level structure corresponding to c2.ula can be: c2:ROW<'uba'STRING, 'ula'STRING, 'uan'STRING>, and the first data structure corresponding to c2.ula is: under the ROW type c2 field, there are STRING type uba, ula, and uan fields.

[0078] Step 130: Obtain the second-level structure information of the reconstructed table structure.

[0079] The second hierarchical structure information includes second field information corresponding to the field to be queried, and a second data structure associated with the second field information.

[0080] As mentioned earlier, since the reconstructed table structure only includes the hierarchical structure of the fields to be queried, the second-level structure information is the hierarchical structure of the fields to be queried. Therefore, the second field information is used to represent the type and name of each field included in the second-level structure. To facilitate understanding of the difference between the second-level structure and the aforementioned first-level structure, the second-level structure of a field is illustrated below.

[0081] For example, if there are multiple fields to be queried, namely c4.su3.ubs.um, c4.su3.ubs.uf, c2.ula, and c4.su2, and the first-level structure corresponding to the fields to be queried, c4.su3.ubs.um, c4.su3.ubs.uf, and c4.su2 is the same as the first-level structure in step 110, then:

[0082] c4:ROW<'su3'ROW<'ubs'ROW<'um'STRING, 'uf'STRING, 'udr'STRING>, 'ut'STRING>, 'su2'ROW<'utm'STRING, 'ujm'STRING>>;

[0083] The first-level structure corresponding to c2.ula is the same as the first-level structure in step 110, which is:

[0084] c2:ROW<'uba'STRING, 'ula'STRING, 'uan'STRING>.

[0085] The second-level structure corresponding to the fields c4.su3.ubs.um, c4.su3.ubs.uf, and c4.su2 to be queried is as follows:

[0086] c4:ROW<'su3'ROW<'ubs'ROW<'um'STRING, 'uf'STRING>>, 'su2'ROW<'utm'STRING, 'ujm'STRING>>;

[0087] The second-level structure corresponding to the field c2.ula to be queried is:

[0088] c2:ROW<'ula'STRING>.

[0089] As seen in the examples above, compared to the first-level structure, the second-level structure only contains the structure related to the field to be queried; the structures of other fields that are not the field to be queried have been filtered out. For example, the c4 field only contains the um and uf fields from the ubs field under the su3 field, as well as the su2 field, but lacks the ut field under the su3 field and the udr field from the ubs field; the c2 field only contains the ula field, but lacks the uba and uan fields.

[0090] Step 140: Obtain the index information of the field to be queried in the second hierarchical structure based on the second field information. In specific implementation, the index information is the subscript information.

[0091] In one embodiment, it can be obtained as follows:

[0092] Step a: Based on the second field information, rename the field to be queried in the second hierarchical structure information to determine the second field name of the field to be queried.

[0093] It is understandable that the second-level structure only includes the hierarchical structure of the field to be queried. Therefore, specifically, the field to be queried can be renamed based on the level of the field to be queried in the second-level structure and the field name of each level in that hierarchical structure. That is, the field to be queried can be renamed based on the second-level structure information, so that the position of the field to be queried in the second-level structure can be directly located based on the name, and the field to be queried can be directly read.

[0094] Step b: In the second hierarchical structure, determine the subscript information corresponding to the second field name of each of the fields to be queried.

[0095] It is understood that the renamed name of the field to be queried is based on the second-level structure information, which includes the level of the field to be queried in the second-level structure and the field information contained in each level. Therefore, the index information of the field to be queried in the second-level structure can be determined based on the renamed name.

[0096] Step 150: Read the field to be queried in the reconstructed table structure based on the index information.

[0097] For example, in one embodiment, the field to be queried can be read from the data structure corresponding to the reconstructed table structure in the table structure of the data storage framework based on the index information.

[0098] The data storage framework is the same as the aforementioned computing engine. In one embodiment, the data storage framework can be Iceberg.

[0099] It is understood that the table structure of the data storage framework can be a table structure generated based on the reconstructed table structure, which facilitates querying using query statements. Therefore, the position of the field to be queried in the table structure of the data storage framework is the same as its position in the reconstructed table structure. That is, based on the above-mentioned index information, the field to be queried can be retrieved from the data structure corresponding to the reconstructed table structure in the table structure of the data storage framework.

[0100] The method for reading nested fields provided by this invention reconstructs a table structure containing only the hierarchical structure of the query field from the first-level structure information of the original table structure corresponding to the query field in the computing engine. Based on the second-level structure information of the query field in the reconstructed table structure, the query field is renamed, and the index information of the query field in the second-level structure information is determined based on the renamed name. Thus, the value of the corresponding query field can be directly read from the table structure of the data storage framework generated based on the reconstructed table structure based on the index information. This eliminates the need to query useless fields and perform data extraction and transformation operations to read nested fields, thereby improving field query performance and saving computing resources.

[0101] In one embodiment, renaming the field to be queried based on the second hierarchical structure information to determine the second field name of the field to be queried includes:

[0102] Obtain the name of the first field corresponding to each field in the second data structure;

[0103] When the field to be queried is a nested field in the first data structure, the field to be queried is renamed based on the first field name to obtain the second field name of the field to be queried.

[0104] It is understandable that, in order to avoid duplicate names after renaming and to facilitate the subsequent determination of the position of the field to be queried based on the renamed name, the field to be queried can be renamed based on the first field name corresponding to each field in the second data structure.

[0105] Specifically, the second field name of the queried field after renaming can be obtained by concatenating the first field names corresponding to each field in the second data structure using a concatenation operator. For ease of understanding, the renaming process is illustrated below.

[0106] For example, if there are multiple fields to be queried, namely c4.su3.ubs.um, c4.su3.ubs.uf, c2.ula, and c4.su2, where the second-level structures corresponding to c4.su3.ubs.um, c4.su3.ubs.uf, and c4.su2 are all:

[0107] c4:ROW<'su3'ROW<'ubs'ROW<'um'STRING, 'uf'STRING>>, 'su2'ROW<'utm'STRING, 'ujm'STRING>>;

[0108] The second-level structure corresponding to the field c2.ula to be queried is:

[0109] c2:ROW<'ula'STRING>;

[0110] Therefore, the second data structure corresponding to c4.su3.ubs.um is as follows: under a field of type ROW, there is a field of type ROW named su3, and under the su3 field, there is a field of type ROW named ubs. The ubs field includes um of type STRING. When renaming the field um to be queried, we can determine that the fields in the outer structure of um are all of type ROW. Therefore, we can determine that the field um to be queried is a nested field, and the first field name of each field in its corresponding nested structure is c4, su3, ubs, and um, respectively. Therefore, the second field name can be obtained as c4-su3-ubs-um. Similarly, the renamed names of c4.su3.ubs.uf, c2.ula, and c4.su2 can be obtained as c4-su3-ubs-uf, c2-ula, and c4-su2, respectively.

[0111] It is understandable that if the field to be queried is not a nested field, for example, if the second level structure of a certain field 'a' is 'a'STRING, meaning that field 'a' is not a nested field, then 'a' can be directly used as the name of the second field.

[0112] In a preferred embodiment, determining the index information corresponding to the second field name of each of the queried fields in the second hierarchical structure includes:

[0113] Determine each first field name contained in the second field name of the field to be queried, query the position information corresponding to each first field name in the second data structure, and obtain an array composed of multiple position information;

[0114] The array of location information is used as the index information of the field to be queried in the second hierarchical structure.

[0115] For example, combining with the previous example, the second-level structure is: c4:ROW<'su3'ROW<'ubs'ROW<'um'STRING, 'uf'STRING>>, 'su2'ROW<'utm'STRING, 'ujm'STRING>>. The corresponding second data structure is: under the ROW type field c4, there are ROW type fields su3 and su2. Under the su3 field, there is a ROW type field ubs, where the ubs field includes STRING type fields um and uf. Under the su2 field, there are STRING type fields um and uf. As can be seen from the second-level structure and the second data structure, there are three more layers outside the layer where the query field c4-su3-ubs-um is located, that is, there are four layers in total. The first field name corresponding to each field is c4, su3, ubs, and um, respectively. Therefore, by finding the relative position information of c4, su3, ubs, and um in the second-level structure, we can obtain the array composed of the position information of each field corresponding to the query field c4-su3-ubs-um, which is [0, 0, 0, 0]. Similarly, we can obtain the subscript information of the query fields c4.su3.ubs.uf, c2.ula, and c4.su2, which are [0, 0, 0, 1], [0, 1], and [1, 0], respectively.

[0116] As mentioned earlier, the position of the field to be queried in the table structure of the data storage framework is the same as its position in the reconstructed table structure. Therefore, the field to be queried can be read from the table structure of the data storage framework based on the above index information.

[0117] In one embodiment, reconstructing the first hierarchical structure information to generate a reconstructed table structure includes:

[0118] Based on the original table structure where the field to be queried is located, determine the type of the outer field corresponding to the field to be queried;

[0119] When the type of the outer field determines that the field to be queried is a nested field, the field in the first-level structure corresponding to the field to be queried is encapsulated as custom data; when the type of the outer field determines that the field to be queried is not a nested field, the custom data is the field to be queried itself.

[0120] The reconstructed table structure is generated based on the custom data.

[0121] It is understood that by saving all fields in the first hierarchical structure as custom data according to the first preset rule based on the first hierarchical structure information, the hierarchical structure information corresponding to the field to be queried can be filtered out from the original table structure corresponding to the calculation engine to obtain the reconstructed table structure, which makes it easier to read the corresponding field to be queried from the reconstructed table structure in the future.

[0122] The first preset rule is used to extract the nested structure and field information corresponding to the field to be queried from the first-level structure. Custom data includes the nested structure and field information corresponding to the field to be queried. Custom data can be represented, for example, as a Rowfield field. For ease of understanding, an example of custom data is given below.

[0123] For example, if there is one field to be queried, namely c4.su3.ubs.um, and the first-level structure corresponding to the field to be queried, c4.su3.ubs.um, is the same as the first-level structure in step 110, then:

[0124] c4:ROW<'su3'ROW<'ubs'ROW<'um'STRING, 'uf'STRING, 'udr'STRING>, 'ut'STRING>, 'su2'ROW<'utm'STRING, 'ujm'STRING>>;

[0125] The custom data corresponding to the field c4.su3.ubs.um to be queried is: c4:ROW<'su3'ROW<'ubs'ROW<'um'STRING>>>.

[0126] If there are multiple fields to be queried, namely c4.su3.ubs.um and c4.su3.ubs.uf, and the first-level structure corresponding to the fields to be queried, c4.su3.ubs.um and c4.su3.ubs.uf, is the same as the first-level structure in step 110, which is:

[0127] c4:ROW<'su3'ROW<'ubs'ROW<'um'STRING, 'uf'STRING, 'udr'STRING>, 'ut'STRING>, 'su2'ROW<'utm'STRING, 'ujm'STRING>>;

[0128] The custom data corresponding to the fields c4.su3.ubs.um and c4.su3.ubs.uf to be queried is: c4:ROW<'su3'ROW<'ubs'ROW<'um'STRING,'uf'STRING>>>.

[0129] Based on any of the above embodiments, in this embodiment, saving all fields in the first hierarchical structure as custom data according to a first preset rule based on the first hierarchical structure information includes:

[0130] The type of the outer field corresponding to the query field is determined based on the original table structure of the calculation engine where the query field is located, and whether the query field is a nested field is determined based on the type of the outer field corresponding to the query field.

[0131] If the field to be queried is determined to be a nested field, the field in the first-level structure corresponding to the field to be queried is encapsulated as custom data; if the field to be queried is determined not to be a nested field, the custom data is the field to be queried itself.

[0132] To facilitate understanding, let's explain how to determine if a queried field is a nested field, using the original table structure corresponding to the calculation engine shown above. Taking c4.su3.ubs.um as an example, we can illustrate the process of determining the type of the queried field. From the original table structure corresponding to the calculation engine, the outer position c4 corresponding to um is of type ROW, which indicates that um is a nested field. In general, if the outer field corresponding to the queried field is of type ROW, it can be understood that the queried field is a nested field.

[0133] Specifically, encapsulating the fields in the first-level structure corresponding to the field to be queried into custom data includes: recursively finding the name and type of the field to be queried from the outside in, based on the type of each field in the first-level structure corresponding to the field to be queried, and encapsulating it into custom data. For ease of understanding, the formation process of the above custom data is illustrated by referring to the original table structure corresponding to the computing engine shown above.

[0134] Taking the field to be queried as c4.su3.ubs.um as an example, from the original table structure corresponding to the above calculation engine, we can obtain that the outer position c4 corresponding to um is of type ROW. Recursively inward, under the ROW type field c4, there is a ROW type su3 field, and under the su3 field, there is a ROW type ubs field, where the ubs field includes the STRING type um. Therefore, the corresponding custom data is: 0={FieldsDataType@14532}”ROW<'su3'ROW<'ubs'ROW<'um'ST "RING>>>", where 0 is the identifier of the field to be queried, c4.su3.ubs.um, and {FieldsDataType@14532} can be understood as the corresponding storage location, which can be ignored; "ROW<'su3'ROW<'ubs'ROW<'um'STRING>>>" represents the custom data corresponding to the field to be queried, c4.su3.ubs.um. Similarly, combining the original table structure corresponding to the calculation engine given in the example above, the custom data corresponding to c4.su3.ubs.uf, c2.ula, and c4.su2 can be obtained one by one as follows:

[0135] 1={FieldsDataType@14533}"ROW<'su3'ROW<'ubs'ROW<'uf'STRING>>>",

[0136] 2={FieldsDataType@14534}"ROW<'su2'ROW<'utm'STRING,'ujm'STRING>>",

[0137] 3={FieldsDataType@14535}"ROW<'ula'STRING>",

[0138] In this context, 1, 2, and 3 are the identifiers of the fields to be queried, c4.su3.ubs.uf, c2.ula, and c4.su2, respectively. ROW<'su3'ROW<'ubs'ROW<'uf'STRING>>>, ROW<'su2'ROW<'utm'STRING, 'ujm'STRING>>, and ROW<'ula'STRING> are the custom data of the fields to be queried, c4.su3.ubs.uf, c2.ula, and c4.su2, respectively.

[0139] Based on any of the above embodiments, in this embodiment, the step of reconstructing the reconstructed table structure according to the custom data includes:

[0140] Based on the type of each field in each level of the first hierarchical structure, determine whether each field is a nested type field;

[0141] If a field of the same nested type in a given layer has multiple fields, the multiple fields and their corresponding field types in the custom data are merged and stored in the layer containing the multiple fields, thus obtaining the reconstructed table structure.

[0142] As can be understood, the first-level structure includes the names and types of all fields, including the field to be queried. Therefore, combining the method for determining nested types of fields described earlier, multiple fields under the same nested type in the custom data are merged and stored in the layer containing these multiple fields. To facilitate understanding, examples from the previous description illustrate the relevant content of merging and storing nested type fields.

[0143] Based on the custom data corresponding to c4.su3.ubs.um, c4.su3.ubs.uf, c2.ula, and c4.su2 given in the above embodiments, it can be seen that in the layer where ubs is located, ubs is a nested type field, and under the ubs field are two fields, um and uf. Therefore, the um and uf fields can be merged and stored under the ubs field. Similarly, since c4 is also a type field, and the c4 field has two fields, su3 and su2, su2 and su3 can be merged and stored under the layer where the c4 field is located. For c2.ula, since there are no multiple fields under the same nested type field in the custom data layer, the original structure can be directly retained. Finally, the reconstructed table structure obtained based on the custom data is as follows:

[0144] |--c4:ROW<'su3'ROW<'ubs'ROW<'um'STRING, 'uf'STRING>>, 'su2'ROW<'utm'STRING, 'ujm'STRING>>,

[0145] |--c2:ROW<'ula'STRING>,

[0146] As can be seen, compared with the original table structure corresponding to the aforementioned calculation engine, the reconstructed table structure is more concise, including only the relevant structure and fields of the field to be queried.

[0147] The method for reading nested fields provided by this invention obtains a reconstructed table structure by reconstructing the table structure. The reconstructed table structure contains only information related to the field to be queried, which facilitates the subsequent reading of the corresponding field to be queried from the reconstructed table structure, while avoiding reading other redundant fields, thereby improving field query performance and saving computing resources.

[0148] To deepen the understanding of the differences between this invention and related technologies, the following will be combined with... Figures 2 to 4 A schematic diagram further illustrates the framework of the related technologies and the field reading method for nested fields provided by this invention. Among them, Figure 2 This is a schematic diagram of the framework flow corresponding to the field reading method of nested fields commonly used in related technologies. Figure 3 This is a diagram illustrating the hierarchical structure corresponding to the field to be queried. Figure 4 A schematic diagram of the framework flow corresponding to the field reading method of nested fields provided by the present invention.

[0149] like Figure 2 As shown, the field reading method for nested fields in related technologies includes the following steps:

[0150] Step 210: Obtain the original table structure TableSchema corresponding to the computing engine.

[0151] The original table structure TableSchema corresponding to the computing engine includes the hierarchical structure information of the fields to be queried and the hierarchical structure information of the fields not to be queried.

[0152] Step 220: Generate the corresponding data storage framework table structure Iceberg Schema based on the original table structure TableSchema corresponding to the computing engine.

[0153] Step 230: Obtain all field information in the hierarchical structure corresponding to the field to be queried from the Iceberg Schema, and obtain the value of the field to be queried from all field information.

[0154] To facilitate a more vivid understanding, such as Figure 2 The present invention provides a schematic diagram of the hierarchical structure corresponding to the field to be queried, as shown in the related technologies for reading nested fields. Figure 3 As shown, Figure 3 The example shows the nested field um under the su3 field under the ubs field under the c4 field. Normally, it is necessary to read all the fields under the c4 field and then extract the um field from all the fields. That is, in addition to reading the um field, it is also necessary to read other fields under c4. The um field cannot be read directly.

[0155] like Figure 4 As shown, the method for reading nested fields provided by this invention includes the following steps:

[0156] Step 410: Obtain the first-level structure information containing the fields to be queried from the original table structure TableSchema corresponding to the computing engine Flink. The first-level structure information can be found in the previous description; for brevity, it will not be repeated here.

[0157] Step 420: Reconstruct the table structure PrunedTableSchema based on the first-level structure information. The reconstructed table structure is the trimmed table structure.

[0158] Refactoring TableSchema into PrunedTableSchema involves using recursion, combination, and other methods to generate a data structure that contains only query fields.

[0159] The process iterates through the `projectFields` array, which represents the index position of the queried field in the TableSchema, generating a type for each corresponding queried field. Based on the elements of the `projectField` array, it first finds the outermost position of the current queried field in the TableSchema and obtains its data type `DataType`. If the current field is a non-nested field, it can be returned directly; otherwise, the following operation is performed. Based on the obtained data type `DataType`, it recursively finds the queried field name and field type from the outermost to the innermost, encapsulates it into a `RowField` field, and saves and returns it.

[0160] TableSchema stipulates that field names at the same level cannot be duplicated; duplicate structures must be merged into a single data structure. If multiple fields at the same level have nested types, they need to be combined into a new RowField and attached to the original level. In the example above, the ubs field contains RowFields...<um STRING> Row<uf STRING> If there are two fields, then these two fields need to be merged into one field, Row.<um STRING,uf STRING> And it's mounted under the `ubs` field. During this merge process, based on the data type of each field obtained in the first step, it's necessary to recursively determine whether the fields at the current level need to be merged, working from the outside in. If no merging is needed, return directly. Otherwise, merge according to the logic described above.

[0161] Through the above two steps, the reconstruction of TableSchema into PrunedTableSchema is completed. The reconstructed PrunedTableSchema only contains the required fields.

[0162] Step 430: Determine the renamed name of the corresponding field to be queried and the position information of the field to be queried in the reconstructed table structure based on the reconstructed table structure PrunedTableSchema.

[0163] Although the data read by Iceberg based on the reconstructed PrunedTableSchema only contains the fields queried, its structure is still a nested structure. At this point, it is necessary to flatten the nested structure, encapsulate the data of the queried fields into RowData, and then distribute it.

[0164] Flattening nested fields requires a schemaIndex array, which specifies the index of each query field within the retrieved nested field data. Iceberg reads data from PrunedTableSchema, so the index of the query field should also be obtained from PrunedTableSchema. Based on the generated projectedFieldNames information, the designed algorithm finds the index of the field in PrunedTableSchema and encapsulates it into a two-dimensional array for display. In practice, this mainly involves determining the nesting level of the current field, then retrieving the field's position at that level based on the field name. When entering a nesting level, it's also necessary to record the position of the outer structure of the current field within the PrunedTableSchema.

[0165] Finally, the data is flattened and distributed. This type is also consistent with the re-specified data output type, outTypeInformation.

[0166] It is understood that the position of the field to be queried can be directly determined based on the location information, i.e., the subscript information mentioned above, so that the corresponding field to be queried can be read directly based on the location information.

[0167] Step 440: Generate the corresponding data storage framework table structure Iceberg Schema based on the reconstructed table structure PrunedTableSchema.

[0168] It can be understood that the position of the field to be queried in the table structure of the data storage framework corresponds to the position of the field to be queried in the reconstructed table structure.

[0169] Step 450: Based on the location information of the field to be queried in the reconstructed table structure, directly read the corresponding field to be queried from the table structure Iceberg Schema of the data storage framework.

[0170] It is understandable that, since the position of the data to be queried in the table structure of the data storage framework can be determined based on the position information of the field to be queried in the reconstructed table structure, the corresponding field to be queried can be directly read from the table structure of the data storage framework based on the position information of the field to be queried in the reconstructed table structure. (Similar to...) Figure 2 Compared to the related methods shown, this method does not require obtaining information other than the field to be queried; it can directly obtain the value corresponding to the field to be queried, thereby improving field query performance and saving computing resources. The effects of the nested field reading method provided by this invention are explained in detail below.

[0171] Specifically, the nested field reading method provided by this invention can achieve benefits in terms of CPU, network bandwidth load, and task execution time when reading Iceberg nested fields. Tests have shown that, under the same conditions, the nested field reading method provided by this invention is superior to... Figure 2 Compared to related methods, CPU utilization is reduced by 20-30%, and the time for Flink to batch read Iceberg fields is also reduced by 20-30%. Furthermore, in terms of network bandwidth load, the benefits increase with the complexity of the nested structure. It is understandable that the main reason for the reduced network bandwidth load is that in related methods, Iceberg reads all data from the currently nested fields and then performs extraction and transformation operations. In the nested field reading method provided in this invention, Iceberg only reads the queried fields, ignoring other fields, thereby reducing network bandwidth load.

[0172] contrast Figure 4 and Figure 2 As can be seen, the nested field reading method provided by this invention can directly read the corresponding field to be queried without reading other fields in the hierarchical structure corresponding to the field to be queried.

[0173] Figure 5 This is a flowchart illustrating an embodiment of a method for reading nested fields provided by the present invention. Figure 5 This can be understood as treating the processing on the computing engine side, the processing on the data storage framework side, and the processing after reading the nested type of query field as a whole, and describing the field reading of the nested field in this invention using this whole.

[0174] like Figure 5As shown, the method for reading nested fields provided by the present invention includes the following steps 510 to 550, wherein steps 510 to 520 can be understood as the processing on the computing engine side, steps 530 to 540 can be understood as the processing on the data storage framework side, and step 550 is the processing after reading the nested type of query field.

[0175] Step 510: Based on the first-level structural information of the fields to be queried in the original table structure TableSchema corresponding to the computing engine, the reconstructed table structure PrunedTableSchema is obtained. The reconstructed table structure is the trimmed table structure.

[0176] The first hierarchical structure information includes the first field information and the first data structure of the field in the first hierarchical structure of the field to be queried.

[0177] The first field information is used to represent the type projectedFieldTypes and the name projectedFieldNames corresponding to each field included in the first hierarchical structure.

[0178] Step 520: Based on the first field information, re-specify the field type and rename the field name, and based on the field name projectedFieldNames in the first field information and the reconstructed table structure PrunedTableSchema, obtain the subscript position information of the field to be queried in the reconstructed table structure PrunedTableSchema.

[0179] Step 530: Based on the reconstructed table structure PrunedTableSchema obtained in step 510, generate the corresponding data storage framework table structure Iceberg Schema.

[0180] Step 540: Based on the ScanContext containing the text field to be queried and the index position information, read the corresponding field to be queried from the Iceberg Schema table structure of the data storage framework.

[0181] Step 550: Based on the index position information, process the field to be queried into a non-nested field for other data processing.

[0182] Specifically, the field to be queried is processed into a non-nested field based on the subscript position information. The subscript information of the field to be queried in the corresponding second-level structure obtained in the above embodiment can be used as the index of the field to be queried, so as to directly locate the position of the corresponding field to be queried and thus process the field to be queried into a non-nested field.

[0183] The field reading device for nested fields provided by the present invention will be described below. The field reading device for nested fields described below can be referred to in correspondence with the field reading method for nested fields described above.

[0184] Figure 6 This is a schematic diagram of the field reading device for nested fields provided by the present invention, as shown below. Figure 6 As shown, the field reading device for nested fields provided by the present invention includes:

[0185] The first acquisition module 610 is used to acquire first-level structure information from the original table structure corresponding to the computing engine; wherein, the first-level structure information includes first field information corresponding to the field to be queried, and a first data structure associated with the first field information; the first data structure is a nested structure.

[0186] The reconstruction module 620 is used to reconstruct the first-level structure information and generate a reconstructed table structure.

[0187] The second acquisition module 630 is used to acquire the second-level structure information of the reconstructed table structure; wherein, the second-level structure information includes second field information corresponding to the field to be queried, and a second data structure associated with the second field information; the second data structure is a nested structure;

[0188] The third acquisition module 640 is used to obtain the index information of the field to be queried in the second hierarchical structure based on the second field information. In specific implementations, the index information can be subscript information.

[0189] The reading module 650 is used to read the field to be queried in the reconstructed table structure based on the index information.

[0190] The nested field reading device provided by this invention obtains a reconstructed table structure containing only the hierarchical structure of the query field by trimming the first-level structure information of the query field in the original table structure corresponding to the computing engine. Based on the second-level structure information of the query field in the reconstructed table structure, the query field is renamed, and the index information of the query field in the second-level structure information is determined based on the renamed name. Thus, the value of the corresponding query field can be directly read from the table structure of the data storage framework generated based on the reconstructed table structure based on the index information. This eliminates the need to query useless fields and perform data extraction and transformation operations to read nested fields, thereby improving field query performance and saving computing resources.

[0191] Figure 7An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can invoke logical instructions in the memory 730 to execute a method for reading nested fields. This method includes: obtaining first-level structure information from the original table structure corresponding to the computing engine; wherein the first-level structure information includes first field information corresponding to the field to be queried, and a first data structure associated with the first field information; the first data structure is a nested structure; reconstructing the first-level structure information to generate a reconstructed table structure; obtaining second-level structure information of the reconstructed table structure; wherein the second-level structure information includes second field information corresponding to the field to be queried, and a second data structure associated with the second field information; the second data structure is a nested structure; obtaining index information of the field to be queried in the second-level structure based on the second field information; and reading the field to be queried in the reconstructed table structure based on the index information.

[0192] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0193] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the nested field reading method provided by the present invention. The method includes: obtaining first-level structure information from the original table structure corresponding to the computing engine; wherein the first-level structure information includes first field information corresponding to the field to be queried, and a first data structure associated with the first field information; the first data structure is a nested structure; reconstructing the first-level structure information to generate a reconstructed table structure; obtaining second-level structure information of the reconstructed table structure; wherein the second-level structure information includes second field information corresponding to the field to be queried, and a second data structure associated with the second field information; the second data structure is a nested structure; obtaining index information of the field to be queried in the second-level structure based on the second field information; and reading the field to be queried in the reconstructed table structure based on the index information.

[0194] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for reading nested fields provided by the present invention. The method includes: obtaining first-level structure information from the original table structure corresponding to the computing engine; wherein the first-level structure information includes first field information corresponding to the field to be queried, and a first data structure associated with the first field information; the first data structure is a nested structure; reconstructing the first-level structure information to generate a reconstructed table structure; obtaining second-level structure information of the reconstructed table structure; wherein the second-level structure information includes second field information corresponding to the field to be queried, and a second data structure associated with the second field information; the second data structure is a nested structure; obtaining index information of the field to be queried in the second-level structure based on the second field information; and reading the field to be queried in the reconstructed table structure based on the index information.

[0195] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0196] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reading nested fields, characterized in that, include: Obtain first-level structure information from the original table structure corresponding to the computing engine; wherein, the first-level structure information includes first field information corresponding to the field to be queried, and a first data structure associated with the first field information; the first data structure is a nested structure; The first-level structural information is reconstructed to generate a reconstructed table structure; Obtain the second-level structure information of the reconstructed table structure; wherein, the second-level structure information only includes the second field information corresponding to the field to be queried, the structure of fields other than the field to be queried has been filtered out, and the second data structure associated with the second field information; the second data structure is a nested structure; Based on the second field information, obtain the index information of the field to be queried in the second hierarchical structure information; The field to be queried in the reconstructed table structure is read based on the index information.

2. The method for reading nested fields according to claim 1, characterized in that, The step of obtaining the index information of the field to be queried in the second hierarchical structure based on the second field information further includes: Based on the second field information, the field to be queried in the second hierarchical structure information is renamed to determine the second field name of the field to be queried; In the second hierarchical structure, determine the index information corresponding to the second field name of each of the fields to be queried.

3. The method for reading nested fields according to claim 2, characterized in that, The step of renaming the field to be queried in the second hierarchical structure information based on the second field information, and determining the second field name of the field to be queried, includes: Obtain the name of the first field corresponding to each field in the second data structure; When the field to be queried is a nested field in the first data structure, the field to be queried is renamed based on the first field name to obtain the second field name of the field to be queried.

4. The method for reading nested fields according to claim 3, characterized in that, In the second hierarchical structure, the index information corresponding to the second field name of each of the fields to be queried is determined, including: Determine each first field name contained in the second field name of the field to be queried, query the position information corresponding to each first field name in the second data structure, and obtain an array composed of multiple position information; The array of location information is used as the index information of the field to be queried in the second hierarchical structure.

5. The method for reading nested fields according to claim 1, characterized in that, The step of reconstructing the first-level structural information to generate a reconstructed table structure includes: Based on the original table structure where the field to be queried is located, determine the type of the outer field corresponding to the field to be queried; When the type of the outer field determines that the field to be queried is a nested field, the field in the first-level structure corresponding to the field to be queried is encapsulated as custom data; when the type of the outer field determines that the field to be queried is not a nested field, the custom data is the field to be queried itself. The reconstructed table structure is generated based on the custom data.

6. The method for reading nested fields according to claim 5, characterized in that, The step of generating the reconstructed table structure based on the custom data includes: Based on the type of each field in each level of the first hierarchical structure, determine whether each field is a nested type field; If a field of the same nested type in a layer has multiple fields, the multiple fields and their corresponding field types in the custom data are merged and stored in the layer where the multiple fields are located, and the reconstructed table structure is obtained.

7. A field reading device for nested fields, characterized in that, The device includes: The first acquisition module is used to acquire first-level structure information from the original table structure corresponding to the computing engine; wherein, the first-level structure information includes first field information corresponding to the field to be queried, and a first data structure associated with the first field information; the first data structure is a nested structure. The reconstruction module is used to reconstruct the first-level structural information and generate a reconstructed table structure. The second acquisition module is used to acquire the second-level structure information of the reconstructed table structure; wherein, the second-level structure information only includes the second field information corresponding to the field to be queried, the structure of fields other than the field to be queried has been filtered out, and the second data structure associated with the second field information; the second data structure is a nested structure; The third acquisition module is used to acquire the index information of the field to be queried in the second hierarchical structure information based on the second field information; The reading module is used to read the field to be queried in the reconstructed table structure based on the index information.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the field reading method for nested fields as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the field reading method for nested fields as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the field reading method for nested fields as described in any one of claims 1 to 6.

Images