Hudi-based index creation method, device, equipment and medium

Abstract

The application relates to the field of financial technology data processing, and discloses an index creation method based on Hudi, which comprises the following steps: obtaining historical data operation information of a preset Hudi data source, analyzing the historical data operation information to obtain a joint query condition corresponding to each data object in the preset Hudi data source, assembling a joint index field of a corresponding data object according to the joint query condition, extracting a data feature of a data object corresponding to each joint index field, determining an index type of the joint index field according to the data feature, obtaining address information of each data object in the preset Hudi data source, and generating an index column file of the preset Hudi data source according to the joint index field, the index type and the address information of each data object. The application further discloses an index creation device based on Hudi, an electronic device and a computer readable storage medium. The application can improve the data query efficiency in the field of financial data.

Description

Technical Field

[0001] This invention relates to the field of financial technology data processing, and in particular to a Hudi-based index creation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] The business modules of financial systems such as banks, securities firms, trusts, insurance companies, funds, and financial leasing companies generate a massive amount of data. For example, the daily peak data for customer group advertising and business settlement can reach tens of millions.

[0003] While indexes can improve the query performance of financial big data, traditional index creation methods have certain drawbacks. For example, index selection can be difficult, and choosing the wrong index or creating too many indexes may lead to performance degradation and resource waste. When queries involve joining multiple tables or complex filtering conditions, index usage may be ineffective, and may even cause the query optimizer to choose an inappropriate execution plan, resulting in decreased query performance. Therefore, the methods for creating indexes for financial big data need further improvement. Summary of the Invention

[0004] This invention provides a Hudi-based index creation method, apparatus, electronic device, and computer-readable storage medium, the main purpose of which is to improve data query efficiency in the field of financial data.

[0005] To achieve the above objectives, this invention provides a Hudi-based index creation method, comprising:

[0006] Obtain historical data operation information from a preset Hudi data source, and parse the historical data operation information to obtain the joint query conditions corresponding to each data object in the preset Hudi data source;

[0007] Assemble the joint index fields of the corresponding data objects according to the joint query conditions;

[0008] Extract the data features of the data object corresponding to each composite index field, and determine the index type of the composite index field based on the data features;

[0009] Obtain the address information of each data object in the preset Hudi data source;

[0010] The preset Hudi data source index column file is generated based on the composite index field, index type, and address information of each data object.

[0011] Optionally, the step of parsing the historical data operation information to obtain the joint query conditions corresponding to each data object in the preset Hudi data source includes:

[0012] According to the preset data manipulation syntax, extract the query condition portion from the historical data manipulation information;

[0013] From the query conditions section, filter out data objects whose query conditions are not unique, and calculate the proportion of data objects whose query conditions are not unique in the historical data operation information;

[0014] Select the query conditions corresponding to data objects whose proportion is greater than the preset proportion threshold as the joint query conditions.

[0015] Optionally, assembling the composite index fields of the corresponding data objects according to the composite query conditions includes:

[0016] Obtain the field set of the data table containing the data object;

[0017] The joint query conditions are segmented into words to obtain a conditional word segmentation set;

[0018] Calculate the similarity between each conditional word segment in the conditional word segmentation set and each field in the field set in turn;

[0019] Fields with similarity greater than a preset similarity threshold are selected as index fields for the corresponding conditional word segmentation;

[0020] By aggregating all the index fields of the data object, we obtain the composite index field of the data object.

[0021] Optionally, extracting the data features of the data object corresponding to each of the composite index fields includes:

[0022] Obtain the value description information of the data object, and the context information of the data object;

[0023] The data object is converted into a data word vector, the value description information is converted into a value vector, and the context information is converted into an associated word vector.

[0024] By concatenating the data word vectors, the value vectors, and the associated word vectors, the data features of the data object are obtained.

[0025] Optionally, determining the index type of the composite index field based on the data characteristics includes:

[0026] The relative probability values ​​between the data features and the preset index type labels are calculated using a pre-trained activation function;

[0027] Calculate the score for each preset index type label based on the relative probability value, and determine the index type label with the highest score as the index type of the data object.

[0028] To address the above problems, the present invention also provides a Hudi-based index creation apparatus, the apparatus comprising:

[0029] The composite index determination module is used to obtain historical data operation information of a preset Hudi data source, parse the historical data operation information to obtain the composite query conditions corresponding to each data object in the preset Hudi data source, and assemble the composite index fields of the corresponding data objects according to the composite query conditions.

[0030] The index type allocation module is used to extract the data characteristics of the data object corresponding to each composite index field, and determine the index type of the composite index field based on the data characteristics;

[0031] The index address acquisition module is used to acquire the address information of each data object in the preset Hudi data source;

[0032] The index file generation module is used to generate the index column file of the preset Hudi data source based on the composite index field, index type and address information of each data object.

[0033] Optionally, the composite index determination module obtains the composite query conditions corresponding to each data object in the preset Hudi data source through the following method:

[0034] According to the preset data manipulation syntax, extract the query condition portion from the historical data manipulation information;

[0035] From the query conditions section, filter out data objects whose query conditions are not unique, and calculate the proportion of data objects whose query conditions are not unique in the historical data operation information;

[0036] Select the query conditions corresponding to data objects whose proportion is greater than the preset proportion threshold as the joint query conditions.

[0037] Optionally, the index type allocation module determines the index type of the composite index field using the following method:

[0038] The relative probability values ​​between the data features and the preset index type labels are calculated using a pre-trained activation function;

[0039] Calculate the score for each preset index type label based on the relative probability value, and determine the index type label with the highest score as the index type of the data object.

[0040] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0041] Memory, storing at least one computer program; and

[0042] The processor executes the program stored in the memory to implement the Hudi-based index creation method described above.

[0043] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the Hudi-based index creation method described above.

[0044] This invention provides a method for parsing historical data operation information to obtain the composite index field corresponding to each data object in a preset Hudi data source. Using composite indexes can improve the retrieval speed and query efficiency of financial big data. Furthermore, obtaining the composite index field from historical data operation information ensures that the selected composite index field is consistent with the actual operations of financial big data, which is beneficial for improving the query efficiency of financial big data. Simultaneously, determining the index type of the corresponding composite index field based on the data characteristics of the data object facilitates the rapid location of the corresponding composite index field in the index column file during actual financial big data queries, based on the index type. Then, the corresponding data object can be obtained based on the address information corresponding to the composite index field. Therefore, the Hudi-based index creation method provided by this invention can improve the query efficiency of financial big data. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating a Hudi-based index creation method according to an embodiment of the present invention.

[0046] Figure 2 This is a detailed implementation flowchart of one step of the Hudi-based index creation method provided in an embodiment of the present invention;

[0047] Figure 3 A detailed implementation flowchart of one step of the Hudi-based index creation method provided in an embodiment of the present invention.

[0048] Figure 4 A functional block diagram of a Hudi-based index creation device provided in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the Hudi-based index creation method according to an embodiment of the present invention.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0052] This application provides a Hudi-based index creation method. The execution entity of the Hudi-based index creation method includes, but is not limited to, at least one of the following: a server, a terminal, or other electronic devices that can be configured to execute the method provided in this application. In other words, the Hudi-based index creation method can be executed by software or hardware installed on a terminal device or a server device, where the software can be a blockchain platform. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0053] Reference Figure 1 The diagram shown illustrates a flowchart of a Hudi-based index creation method according to an embodiment of the present invention. In this embodiment, the Hudi-based index creation method includes:

[0054] S1. Obtain historical data operation information of a preset Hudi data source, and parse the historical data operation information to obtain the joint query conditions corresponding to each data object in the preset Hudi data source;

[0055] In this embodiment of the invention, the preset Hudi data source refers to the storage area of ​​financial big data that uses Hudi technology for data management. The storage area includes, but is not limited to, databases, blockchains, network caches, etc.

[0056] In this embodiment of the invention, the historical data operation information refers to the set of data operation instructions such as data query, data addition, data insertion, data modification, and data deletion that occurred in the preset Hudi data source within a historical time period. The historical time period can be set according to the actual data volume of the preset Hudi data source and the frequency and number of data operations, and can be set to the most recent six months or three months.

[0057] In this embodiment of the invention, the data object refers to the data field of the final operation corresponding to each data operation instruction in the historical data operation information, such as the field to be added, the field to be deleted, or the field to be searched.

[0058] Understandably, data retrieval efficiency is relatively high when a single query condition can be used to locate data objects, while it is relatively low when multiple query conditions are layered and filtered to locate data objects. For example, when searching for the policy term of a user whose insurance type is life insurance and who has a claim record, the policy term is the final data object to be queried. The corresponding filter conditions for this data object are that the insurance type must be life insurance and the corresponding life insurance claim record is greater than 0.

[0059] In this embodiment of the invention, the historical data operation information can be subjected to syntactic analysis to obtain the joint query conditions corresponding to each data object.

[0060] For details, please refer to Figure 2 As shown, the step of parsing the historical data operation information to obtain the joint query conditions corresponding to each data object in the preset Hudi data source includes:

[0061] S11. Extract the query condition portion from the historical data operation information according to the preset data operation syntax;

[0062] S12. From the query condition section, filter out data objects whose query conditions are not unique, and count the proportion of data objects whose query conditions are not unique in the historical data operation information.

[0063] S13. Select the query conditions corresponding to the data objects whose proportion is greater than the preset proportion threshold as the joint query conditions.

[0064] In this embodiment of the invention, the preset data operation syntax can be determined according to the data operation language used by the actual preset Hudi data source. For example, the preset data operation syntax can be selected as SQL syntax.

[0065] In this embodiment of the invention, the preset proportion threshold can be set according to the actual situation of financial big data. The purpose of selecting the query conditions corresponding to data objects with a proportion greater than the preset proportion threshold as the joint query conditions is to ensure the necessity of setting a joint index for the data objects.

[0066] This invention provides a way to obtain joint query conditions from historical data operation information in order to filter out data objects suitable for setting up joint indexes.

[0067] S2. Assemble the joint index fields of the corresponding data objects according to the joint query conditions;

[0068] It is understood that the joint query conditions include at least two or more query restrictions, and all query conditions are ANDed, meaning that no single query condition can determine the final data object.

[0069] For example, to find the user ID of a male customer who has purchased both life insurance and auto insurance, the data object to be queried is the user ID, and the joint query conditions corresponding to the user ID include the insurance type being life insurance and auto insurance, and the gender corresponding to the user ID being male.

[0070] Compared to traditional indexing methods, which require first finding a first data object whose index field is life insurance, then filtering from the first data object using the index field of auto insurance to obtain a second data object, and finally selecting a third data object from the second data object whose index field is male, this data query method is relatively inefficient. To address this, this embodiment of the invention uses a composite index field to quickly locate the third data object.

[0071] For details, please refer to Figure 3 As shown, assembling the composite index fields of the corresponding data objects according to the composite query conditions includes:

[0072] S21. Obtain the field set of the data table where the data object is located;

[0073] S22. Perform word segmentation on the joint query conditions to obtain a conditional word segmentation set;

[0074] S23. Calculate the similarity between each conditional word in the conditional word segmentation set and each field in the field set in turn;

[0075] S24. Select the fields with similarity greater than the preset similarity threshold as the index fields for the corresponding conditional word segmentation;

[0076] S25. Gather all the index fields of the data object to obtain the composite index field of the data object.

[0077] In this embodiment of the invention, the joint query conditions are text content expressed in natural language, and existing word segmentation tools can be used to segment the joint query conditions.

[0078] In this embodiment of the invention, the similarity between each conditional word in the conditional word segmentation set and each field in the field set can be calculated by fuzzy matching. The higher the fuzzy matching degree between each conditional word and each field, the higher the similarity between the corresponding conditional word and the field.

[0079] In another optional embodiment of the present invention, the similarity between the corresponding conditional segment and the field can be determined by calculating the textual semantic similarity between each conditional segment and each field.

[0080] In this embodiment of the invention, the preset similarity threshold can be set according to the actual situation of financial big data.

[0081] S3. Extract the data features of the data object corresponding to each composite index field, and determine the index type of the composite index field based on the data features;

[0082] In this embodiment of the invention, the purpose of setting appropriate index types for different data objects is to improve the retrieval efficiency of the index by setting suitable index types. For example, for fields that are queried within a data range, a binary tree index or a B+ tree index can be used.

[0083] In this embodiment of the invention, an index type is selected based on the data characteristics corresponding to each data object, wherein the data characteristics include, but are not limited to, data value characteristics, data correlation characteristics, etc.

[0084] In this embodiment of the invention, a deep learning-based convolutional neural network model can be used to extract the data features of the data object corresponding to each of the joint index fields.

[0085] Specifically, the extraction of data features of the data object corresponding to each of the composite index fields includes:

[0086] Obtain the value description information of the data object, and the context information of the data object;

[0087] The data object is converted into a data word vector, the value description information is converted into a value vector, and the context information is converted into an associated word vector.

[0088] By concatenating the data word vectors, the value vectors, and the associated word vectors, the data features of the data object are obtained.

[0089] In this embodiment of the invention, models with word vector conversion capabilities, such as word2vec and NLP (Natural Language Processing), can be used to convert the data object into data word vectors, the value description information into value vectors, and the context information into associated word vectors.

[0090] In this embodiment of the invention, the data features are typically in the form of vectors. By calculating the vectors corresponding to the data features, the index type corresponding to the data features can be obtained.

[0091] Specifically, determining the index type of the composite index field based on the data characteristics includes:

[0092] The relative probability values ​​between the data features and the preset index type labels are calculated using a pre-trained activation function;

[0093] Calculate the score for each preset index type label based on the relative probability value, and determine the index type label with the highest score as the index type of the data object.

[0094] In this embodiment of the invention, the pre-trained activation function includes, but is not limited to, the softmax activation function, the sigmoid activation function, and the ReLU activation function, and the preset index type label includes, but is not limited to, Bloom Filter index label, binary tree index label, or B+ tree index label.

[0095] In another optional embodiment of the present invention, the index type of each data object can also be determined by benchmark testing and repeated trials.

[0096] S4. Obtain the address information of each data object in the preset Hudi data source, and generate the index column file of the preset Hudi data source according to the composite index field, index type and address information of each data object.

[0097] In this embodiment of the invention, the address information of each data object in the preset Hudi data source can be obtained using the Parquet file corresponding to each data object. The Parquet file is a columnar storage format file that can be used to efficiently store and process data in a financial big data environment.

[0098] Preferably, the composite index field, index type, and address information of the data object can be encoded using the columnar encoding format of the Parquet file corresponding to each data object. For example, encoding formats such as RunLength Encoding (RLE), Delta Encoding, and Bit Packing can be used to assemble the index column file of the preset Hudi data source.

[0099] The index column file generated based on the Parquet file uses columnar encoding, which can further reduce the storage space of the index file. At the same time, it supports adding, deleting or modifying the definition of columns in the index column file without destroying the existing data, thus improving the flexibility and compatibility of the index column file.

[0100] This invention provides a method for parsing historical data operation information to obtain the composite index field corresponding to each data object in a preset Hudi data source. Using composite indexes can improve the retrieval speed and query efficiency of financial big data. Furthermore, obtaining the composite index field from historical data operation information ensures that the selected composite index field is consistent with the actual operations of financial big data, which is beneficial for improving the query efficiency of financial big data. Simultaneously, determining the index type of the corresponding composite index field based on the data characteristics of the data object facilitates the rapid location of the corresponding composite index field in the index column file during actual financial big data queries, based on the index type. Then, the corresponding data object can be obtained based on the address information corresponding to the composite index field. Therefore, the Hudi-based index creation method provided by this invention can improve the query efficiency of financial big data.

[0101] like Figure 4 The diagram shown is a functional block diagram of an index creation device based on Hudi provided in an embodiment of the present invention.

[0102] The Hudi-based index creation device 100 described in this invention can be installed in an electronic device. Depending on the functions implemented, the Hudi-based index creation device 100 includes a composite index determination module 101, an index type allocation module 102, an index address acquisition module 103, and an index file generation module 104. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0103] In this embodiment, the functions of each module / unit are as follows:

[0104] The composite index determination module 101 is used to obtain historical data operation information of a preset Hudi data source, parse the historical data operation information to obtain the composite query conditions corresponding to each data object in the preset Hudi data source, and assemble the composite index fields of the corresponding data objects according to the composite query conditions.

[0105] The index type allocation module 102 is used to extract the data characteristics of the data object corresponding to each composite index field, and determine the index type of the composite index field based on the data characteristics;

[0106] The index address acquisition module 103 is used to acquire the address information of each data object in the preset Hudi data source;

[0107] The index file generation module 104 is used to generate the index column file of the preset Hudi data source based on the composite index field, index type and address information of each data object.

[0108] In detail, the specific implementation methods of each module of the Hudi-based index creation device 100 are as follows:

[0109] Step 1: Obtain historical data operation information from the preset Hudi data source, and parse the historical data operation information to obtain the joint query conditions corresponding to each data object in the preset Hudi data source;

[0110] In this embodiment of the invention, the preset Hudi data source refers to the storage area of ​​financial big data that uses Hudi technology for data management. The storage area includes, but is not limited to, databases, blockchains, network caches, etc.

[0111] In this embodiment of the invention, the historical data operation information refers to the set of data operation instructions such as data query, data addition, data insertion, data modification, and data deletion that occurred in the preset Hudi data source within a historical time period. The historical time period can be set according to the actual data volume of the preset Hudi data source and the frequency and number of data operations, and can be set to the most recent six months or three months.

[0112] In this embodiment of the invention, the data object refers to the data field of the final operation corresponding to each data operation instruction in the historical data operation information, such as the field to be added, the field to be deleted, or the field to be searched.

[0113] Understandably, data retrieval efficiency is relatively high when a single query condition can be used to locate data objects, while it is relatively low when multiple query conditions are layered and filtered to locate data objects. For example, when searching for the policy term of a user whose insurance type is life insurance and who has a claim record, the policy term is the final data object to be queried. The corresponding filter conditions for this data object are that the insurance type must be life insurance and the corresponding life insurance claim record is greater than 0.

[0114] In this embodiment of the invention, the historical data operation information can be subjected to syntactic analysis to obtain the joint query conditions corresponding to each data object.

[0115] In detail, the step of parsing the historical data operation information to obtain the joint query conditions corresponding to each data object in the preset Hudi data source includes:

[0116] According to the preset data manipulation syntax, extract the query condition portion from the historical data manipulation information;

[0117] From the query conditions section, filter out data objects whose query conditions are not unique, and calculate the proportion of data objects whose query conditions are not unique in the historical data operation information;

[0118] Select the query conditions corresponding to data objects whose proportion is greater than the preset proportion threshold as the joint query conditions.

[0119] In this embodiment of the invention, the preset data operation syntax can be determined according to the data operation language used by the actual preset Hudi data source. For example, the preset data operation syntax can be selected as SQL syntax.

[0120] In this embodiment of the invention, the preset proportion threshold can be set according to the actual situation of financial big data. The purpose of selecting the query conditions corresponding to data objects with a proportion greater than the preset proportion threshold as joint query conditions is to ensure the necessity of setting a joint index for the data objects.

[0121] This invention provides a way to obtain joint query conditions from historical data operation information in order to filter out data objects suitable for setting up joint indexes.

[0122] Step 2: Assemble the composite index fields of the corresponding data objects according to the composite query conditions;

[0123] It is understood that the joint query conditions include at least two or more query restrictions, and all query conditions are ANDed, meaning that no single query condition can determine the final data object.

[0124] For example, to find the user ID of a male customer who has purchased both life insurance and auto insurance, the data object to be queried is the user ID, and the joint query conditions corresponding to the user ID include the insurance type being life insurance and auto insurance, and the gender corresponding to the user ID being male.

[0125] Compared to traditional indexing methods, which require first finding a first data object whose index field is life insurance, then filtering from the first data object using the index field of auto insurance to obtain a second data object, and finally selecting a third data object from the second data object whose index field is male, this data query method is relatively inefficient. To address this, this embodiment of the invention uses a composite index field to quickly locate the third data object.

[0126] Specifically, assembling the composite index fields of the corresponding data objects based on the composite query conditions includes:

[0127] Obtain the field set of the data table containing the data object;

[0128] The joint query conditions are segmented into words to obtain a conditional word segmentation set;

[0129] Calculate the similarity between each conditional word segment in the conditional word segmentation set and each field in the field set in turn;

[0130] Fields with similarity greater than a preset similarity threshold are selected as index fields for the corresponding conditional word segmentation;

[0131] By aggregating all the index fields of the data object, we obtain the composite index field of the data object.

[0132] In this embodiment of the invention, the joint query conditions are text content expressed in natural language, and existing word segmentation tools can be used to segment the joint query conditions.

[0133] In this embodiment of the invention, the similarity between each conditional word in the conditional word segmentation set and each field in the field set can be calculated by fuzzy matching. The higher the fuzzy matching degree between each conditional word and each field, the higher the similarity between the corresponding conditional word and the field.

[0134] In another optional embodiment of the present invention, the similarity between the corresponding conditional segment and the field can be determined by calculating the textual semantic similarity between each conditional segment and each field.

[0135] In this embodiment of the invention, the preset similarity threshold can be set according to the actual situation of financial big data.

[0136] Step 3: Extract the data features of the data object corresponding to each composite index field, and determine the index type of the composite index field based on the data features;

[0137] In this embodiment of the invention, the purpose of setting appropriate index types for different data objects is to improve the retrieval efficiency of the index by setting suitable index types. For example, for fields that are queried within a data range, a binary tree index or a B+ tree index can be used.

[0138] In this embodiment of the invention, an index type is selected based on the data characteristics corresponding to each data object, wherein the data characteristics include, but are not limited to, data value characteristics, data correlation characteristics, etc.

[0139] In this embodiment of the invention, a deep learning-based convolutional neural network model can be used to extract the data features of the data object corresponding to each of the joint index fields.

[0140] Specifically, the extraction of data features of the data object corresponding to each of the composite index fields includes:

[0141] Obtain the value description information of the data object, and the context information of the data object;

[0142] The data object is converted into a data word vector, the value description information is converted into a value vector, and the context information is converted into an associated word vector.

[0143] By concatenating the data word vectors, the value vectors, and the associated word vectors, the data features of the data object are obtained.

[0144] In this embodiment of the invention, models with word vector conversion capabilities, such as word2vec and NLP (Natural Language Processing), can be used to convert the data object into data word vectors, the value description information into value vectors, and the context information into associated word vectors.

[0145] In this embodiment of the invention, the data features are typically in the form of vectors. By calculating the vectors corresponding to the data features, the index type corresponding to the data features can be obtained.

[0146] Specifically, determining the index type of the composite index field based on the data characteristics includes:

[0147] The relative probability values ​​between the data features and the preset index type labels are calculated using a pre-trained activation function;

[0148] Calculate the score for each preset index type label based on the relative probability value, and determine the index type label with the highest score as the index type of the data object.

[0149] In this embodiment of the invention, the pre-trained activation function includes, but is not limited to, the softmax activation function, the sigmoid activation function, and the ReLU activation function, and the preset index type label includes, but is not limited to, Bloom Filter index label, binary tree index label, or B+ tree index label.

[0150] In another optional embodiment of the present invention, the index type of each data object can also be determined by benchmark testing and repeated trials.

[0151] Step 4: Obtain the address information of each data object in the preset Hudi data source, and generate the index column file of the preset Hudi data source based on the composite index field, index type and address information of each data object.

[0152] In this embodiment of the invention, the address information of each data object in the preset Hudi data source can be obtained using the Parquet file corresponding to each data object. The Parquet file is a columnar storage format file that can be used to efficiently store and process data in a financial big data environment.

[0153] Preferably, the composite index field, index type, and address information of the data object can be encoded using the columnar encoding format of the Parquet file corresponding to each data object. For example, encoding formats such as RunLength Encoding (RLE), Delta Encoding, and Bit Packing can be used to assemble the index column file of the preset Hudi data source.

[0154] The index column file generated based on the Parquet file uses columnar encoding, which can further reduce the storage space of the index file. At the same time, it supports adding, deleting or modifying the definition of columns in the index column file without destroying the existing data, thus improving the flexibility and compatibility of the index column file.

[0155] This invention application parses historical data operation information to obtain the composite index field corresponding to each data object in a preset Hudi data source. Using composite indexes can improve the retrieval speed and query efficiency of financial big data. Furthermore, obtaining composite index fields from historical data operation information ensures that the selected composite index fields are consistent with the actual operations of financial big data, which is beneficial for improving the query efficiency of financial big data. Simultaneously, determining the index type of the corresponding composite index field based on the data characteristics of the data object facilitates the rapid location of the corresponding composite index field in the index column file based on the index type during actual financial big data queries. Then, the corresponding data object can be obtained based on the address information corresponding to the composite index field. Therefore, the Hudi-based index creation device provided by this invention application can improve the query efficiency of financial big data.

[0156] like Figure 5 The diagram shown is a schematic representation of an electronic device that implements a Hudi-based index creation method according to an embodiment of the present invention.

[0157] The electronic device 1 may include a processor 10, a memory 11, and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as Hudi-based index creation.

[0158] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 1. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as code created based on Hudi indexes, but also to temporarily store data that has been output or will be output.

[0159] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., Hudi-based index creation) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0160] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0161] Figure 5 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 5The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0162] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0163] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0164] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0165] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0166] The Hudi-based index creation stored in the memory 11 of the electronic device 1 is a combination of multiple instructions that, when run in the processor 10, can be implemented as follows:

[0167] Obtain historical data operation information from a preset Hudi data source, and parse the historical data operation information to obtain the joint query conditions corresponding to each data object in the preset Hudi data source;

[0168] Assemble the joint index fields of the corresponding data objects according to the joint query conditions;

[0169] Extract the data features of the data object corresponding to each composite index field, and determine the index type of the composite index field based on the data features;

[0170] Obtain the address information of each data object in the preset Hudi data source;

[0171] The preset Hudi data source index column file is generated based on the composite index field, index type, and address information of each data object.

[0172] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0173] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0174] Obtain historical data operation information from a preset Hudi data source, and parse the historical data operation information to obtain the joint query conditions corresponding to each data object in the preset Hudi data source;

[0175] Assemble the joint index fields of the corresponding data objects according to the joint query conditions;

[0176] Extract the data features of the data object corresponding to each composite index field, and determine the index type of the composite index field based on the data features;

[0177] Obtain the address information of each data object in the preset Hudi data source;

[0178] The preset Hudi data source index column file is generated based on the composite index field, index type, and address information of each data object.

[0179] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0180] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0181] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0182] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0183] The embodiments of this application can acquire and process relevant data based on holographic projection technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0184] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A Hudi-based index creation method, characterized in that, The method includes: Obtain historical data operation information from a preset Hudi data source, calculate the proportion of data objects in the historical data operation information whose query conditions are not unique, and select the query conditions corresponding to data objects whose proportion is greater than a preset proportion threshold as joint query conditions. Obtain the field set of the data table where the data object is located, perform word segmentation on the joint query conditions to obtain the condition word segmentation set, determine the index field based on the similarity between each field in the field set and each condition word in the condition word segmentation set, and gather all the index fields of the data object to obtain the joint index field of the data object; Extract data word vectors, value vectors, and associated word vectors as data features from the data object corresponding to each of the composite index fields. Calculate the relative probability value between the data features and the preset index type labels. Calculate the score of each preset index type label based on the relative probability value. Determine the index type label with the highest score as the index type of the data object. Obtain the address information of each data object in the preset Hudi data source; The preset Hudi data source index column file is generated based on the composite index field, index type, and address information of each data object.

2. The Hudi-based index creation method as described in claim 1, characterized in that, The calculation of the proportion of data objects in the historical data operation information whose query conditions are not unique includes: According to the preset data manipulation syntax, extract the query condition portion from the historical data manipulation information; From the query conditions section, filter out data objects whose query conditions are not unique, and calculate the proportion of data objects whose query conditions are not unique in the historical data operation information.

3. The Hudi-based index creation method as described in claim 1, characterized in that, The process of determining the index field based on the similarity between each field in the field set and each conditional word in the conditional word segmentation set includes: Calculate the similarity between each conditional word segment in the conditional word segmentation set and each field in the field set in turn; Fields with similarity greater than a preset similarity threshold are selected as index fields for the corresponding conditional word segmentation.

4. The Hudi-based index creation method as described in claim 1, characterized in that, The step of extracting data word vectors, value vectors, and associated word vectors as data features from the data object corresponding to each of the composite index fields includes: Obtain the value description information of the data object, and the context information of the data object; The data object is converted into a data word vector, the value description information is converted into a value vector, and the context information is converted into an associated word vector. By concatenating the data word vectors, the value vectors, and the associated word vectors, the data features of the data object are obtained.

5. A Hudi-based index creation device, characterized in that, The device includes: The composite index determination module is used to obtain historical data operation information of a preset Hudi data source, calculate the proportion of data objects in the historical data operation information whose query conditions are not unique in the historical data operation information, and select the query conditions corresponding to data objects whose proportion is greater than a preset proportion threshold as composite query conditions. The composite index determination module is also used to obtain the field set of the data table where the data object is located, perform word segmentation on the composite query conditions to obtain a condition word segmentation set, determine the index field based on the similarity between each field in the field set and each condition word in the condition word segmentation set, and collect all the index fields of the data object to obtain the composite index field of the data object; The index type allocation module is used to extract data word vectors, value vectors and related word vectors as data features from the data object corresponding to each of the composite index fields, calculate the relative probability value between the data features and the preset index type labels, calculate the score of each preset index type label based on the relative probability value, and determine the index type label with the highest score as the index type of the data object. The index address acquisition module is used to acquire the address information of each data object in the preset Hudi data source; The index file generation module is used to generate the index column file of the preset Hudi data source based on the composite index field, index type and address information of each data object.

6. The Hudi-based index creation apparatus as described in claim 5, characterized in that, The composite index determination module uses the following method to statistically determine the proportion of data objects in the historical data operation information that do not have unique query conditions: According to the preset data manipulation syntax, extract the query condition portion from the historical data manipulation information; From the query criteria section, filter out data objects whose query criteria are not unique, and calculate the proportion of data objects whose query criteria are not unique in the historical data operation information.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the Hudi-based index creation method as described in any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the Hudi-based index creation method as described in any one of claims 1 to 4.

Images