Index data processing method and device, index generation method and device, medium and equipment

Abstract

The invention relates to an index data processing method and device, an index generation method and device, a medium and equipment, and the index data processing method comprises the steps: when data is written into a data table, obtaining index data corresponding to target data written into the data table, the index data is data corresponding to an index field used for constructing an index in the target data; based on the log queues corresponding to the data table, determining a target log queue and a target index log file to which each piece of index data needs to be written, a plurality of index log files being stored in each log queue, and a plurality of pieces of index data being stored in each index log file; index data corresponding to the target data are written into the target index log file in the target log queue, and the index log file is a pre-writing type log file and is used for data playback to generate an index corresponding to the index data.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to an index data processing method, an index generation method, a device, a medium and a device. Background Art

[0002] In a database system, an index needs to be generated when data is written so that users can query the data. However, under large scale and high load, the generation and writing of indexes will cause the index writing delay to increase, resulting in user query delays. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a method for processing index data, the method comprising:

[0005] When writing data in a data table, obtaining index data corresponding to the target data written in the data table, wherein the index data is data corresponding to an index field in the target data used to construct an index;

[0006] Based on the log queue corresponding to the data table, determine the target log queue and the target index log file to which each of the index data is to be written, wherein each of the log queues stores a plurality of index log files, and each of the index log files stores a plurality of index data;

[0007] The index data corresponding to the target data is written into the target index log file in the target log queue, wherein the index log file is a write-ahead log file used for data playback to generate an index corresponding to the index data.

[0008] In a second aspect, the present disclosure provides an index generation method, the method comprising:

[0009] Determine a to-be-processed log queue corresponding to an to-be-processed index log file, wherein the index log file is generated based on the index data processing method described in the first aspect;

[0010] Obtaining the index log file from the to-be-processed log queue;

[0011] Data playback is performed based on the index data in the index log file to generate an index corresponding to the index data.

[0012] In a third aspect, the present disclosure provides a device for processing index data, the device comprising:

[0013] A first acquisition module is used to acquire index data corresponding to target data written in the data table when writing data in the data table, wherein the index data is data corresponding to an index field in the target data used to construct an index;

[0014] A first determination module is used to determine a target log queue and a target index log file to be written into each of the index data based on the log queue corresponding to the data table, wherein each of the log queues stores a plurality of index log files, and each of the index log files stores a plurality of index data;

[0015] The first processing module is used to write the index data corresponding to the target data into the target index log file in the target log queue, wherein the index log file is a pre-written log file, and the index log file is used for data playback to generate an index corresponding to the index data.

[0016] In a fourth aspect, the present disclosure provides an index generation device, the device comprising:

[0017] A fourth determination module, used to determine a to-be-processed log queue corresponding to an to-be-processed index log file, wherein the index log file is generated based on the index data processing method described in the first aspect;

[0018] A second acquisition module is used to acquire the index log file from the to-be-processed log queue;

[0019] A generation module is used to perform data playback based on the index data in the index log file and generate an index corresponding to the index data.

[0020] In a fifth aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect or the second aspect.

[0021] In a sixth aspect, the present disclosure provides an electronic device, including:

[0022] a storage device having a computer program stored thereon;

[0023] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect or the second aspect.

[0024] In the above technical solution, when writing data in a data table, the index data corresponding to the target data written in the data table is obtained, and then based on the log queue corresponding to the data table, the target log queue and the target index log file to which each index data is to be written are determined, so that the index data is written into the target index log file in the target log queue. The index log file is used for data playback to generate the index corresponding to the index data. Based on this solution, the index data therein can be obtained and stored when the data is written to ensure the accurate generation of the index, effectively solve the processing efficiency of the index data of large-scale data, and avoid the problem of index loss after power failure. At the same time, in this technical solution, the data writing process can be load balanced based on the log queue during the process of writing the index data into the index log file, so as to provide support for the realization of balanced writing of the index data, ensure the efficiency of the subsequent processing of the index data in the index log file, facilitate the improvement of the efficiency of index generation, and reduce the delay of index writing and user query delay.

[0025] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0027] Figure 1 It is a flowchart of a method for processing index data provided according to an embodiment of the present disclosure.

[0028] Figure 2 The present invention is a block diagram of an apparatus for processing index data provided according to an embodiment of the present invention.

[0029] Figure 3 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0031] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0033] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0035] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0036] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0037] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0038] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0039] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0040] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0041] Figure 1 As shown, it is a flowchart of a method for processing index data according to an embodiment of the present disclosure. Figure 1 As shown, the method may include:

[0042] In step 11, when writing data in the data table, index data corresponding to the target data written in the data table is obtained, wherein the index data is data corresponding to an index field in the target data used to construct an index.

[0043] The data table may be a table in the storage system for writing data, such as a student table and a class table in the education system, which may be pre-set or specified according to the actual application scenario to determine the data table in which the index needs to be generated in real time. The written data is the data written to the data table in the storage system.

[0044] As an example, the index field for building the index can be set according to the actual index needs. For example, if the data table is a student table, in which the fields may include name, student number, class, age, etc., if you need to build an index for the name, you can set the index field to the name field. Accordingly, when writing data into the data table, the data corresponding to the index field therein can be obtained as the index data, that is, the data value under the name field in the written data is obtained. In this step, when the original data is written into the data table in the storage system, the data corresponding to the pre-configured index field can be obtained, so that the index data in the written data can be written into the log file for subsequent index generation, thereby avoiding the problem of index loss due to power failure after the original data is written.

[0045] In step 12, based on the log queue corresponding to the data table, the target log queue and the target index log file to which each index data is to be written are determined, wherein each log queue stores multiple index log files, and each index log file stores multiple index data.

[0046] The log queue is used to store index log files and may also contain metadata for the index log files. In the process of distributed index data processing, data producers can write index data into index log files and store them in corresponding log queues. Data consumers can access corresponding log queues to obtain index log files from the log queues and build indexes based on the index data in the index log files. The number of log queues can be set based on actual application scenarios.

[0047] In this embodiment, there can be multiple data tables and log queues. Accordingly, the log queue corresponding to each data table can be pre-set. For example, the log queue can be allocated according to the data processing volume corresponding to each data table. For example, there are 8 log queues, namely Q0-Q7. In the initial case, log queues Q0-Q4 are allocated to data table T1, log queues Q5 and Q6 are allocated to data table T2, and log queue Q7 is allocated to data tables T3-T6. When writing data in data table T1, index data can be obtained from the written data, and the target log queue can be determined from the log queues Q0-Q4 corresponding to data table T1, and the target index log file under the target log queue can be further determined.

[0048] In step 13, the index data corresponding to the target data is written into the target index log file in the target log queue, wherein the index log file is a write-ahead log (WAL) file, and the index log file is used for data playback to generate the index corresponding to the index data. The WAL file implements a data playback mechanism to ensure that each index data that has not been flushed can be found in the WAL file to ensure the integrity of the index data.

[0049] In this embodiment, after the index data corresponding to the target data written in the data table is extracted, it can be written into the file under the log queue corresponding to the data table, so as to realize the balanced processing of index data between different data tables. And further determine the target index log file to which the index data needs to be written, so as to support the balanced processing of index data in different index log files when the index data is written, that is, to distribute the index data in the production process of the index data.

[0050] Therefore, in the above technical solution, when writing data in the data table, the index data corresponding to the target data written in the data table is obtained, and then based on the log queue corresponding to the data table, the target log queue and the target index log file to be written for each index data are determined, so that the index data is written into the target index log file in the target log queue. The index log file is used for data playback to generate the index corresponding to the index data. Based on this solution, the index data therein can be obtained and stored when the data is written to ensure the accurate generation of the index, effectively solve the processing efficiency of the index data of large-scale data, and avoid the problem of losing the index after power failure. At the same time, in this technical solution, the data writing process can be load-balanced based on the log queue during the process of writing the index data into the index log file, so as to provide support for the realization of balanced writing of the index data, ensure the efficiency of the subsequent processing of the index data in the index log file, facilitate the improvement of the efficiency of index generation, and reduce the delay of index writing and user query delay.

[0051] In a possible embodiment, the exemplary implementation method of determining the target log queue and the target index log file to be written to each index data based on the log queue corresponding to the data table may include:

[0052] A hash operation is performed on the index data to obtain a hash value of the index data.

[0053] The target log queue is determined according to the hash value and the log queue corresponding to the data table.

[0054] Wherein, it can be processed based on the hash method commonly used in the art to obtain the hash value. As an example, the number of log queues corresponding to the data table can be obtained, which is recorded as hashShard, and the target log queue can be determined according to the value obtained by taking the modulus of the number of log queues corresponding to the data table according to the hash value. If the log queues corresponding to the data table are Q0, Q1, and Q2, which can correspond to the modulus values ​​0, 1, and 2 respectively, then the value obtained by taking the modulus of quantity 3 based on the hash value is 1, then Q1 can be used as the target log queue corresponding to the index data, that is, the index data is written to the index log file under the Q1 directory. In this step, the index data is hashed, which is convenient for the balanced distribution of the index data to the log queue for processing on the one hand, and the binding association of the index data and the target log queue can be realized through the hash operation on the other hand, which is convenient for subsequent data playback and query process.

[0055] Furthermore, the target index log file may be determined according to the hash value, the target log queue, and the attribute information corresponding to the index data.

[0056] Among them, each log queue can correspond to a shard, which represents a directory of a group of index log files. There can be multiple index log files under this directory. In this step, the file corresponding to the index data can be further determined from the index log files in the target log queue.

[0057] As an example, the identifier of the target log queue may be encoded in the directory information, and then the query may be performed directly based on the identifier of the target log queue during the query process, thereby improving data query efficiency.

[0058] In a possible embodiment, to facilitate quick query of the index log file, the following information may be encoded in the file name of the index log file:

[0059] tableId, used to indicate the table ID of the data table corresponding to the index data;

[0060] columnId, used to indicate the field identifier of the index field corresponding to the index data;

[0061] hashId / hashShard, where hashShard is used to indicate the number of log queues corresponding to the data table to which the index data belongs, and hashId is used to indicate the value obtained by taking the hash value of the index data modulo the number of log queues. By determining hashId, the index data is evenly written to different log queues, thereby improving the load balancing of index data processing;

[0062] UUID can be implemented by using random numbers without duplication, thus ensuring that the names of various index log files will not conflict.

[0063] If a time series index is to be constructed, the file name may further include timeBucket, which is used to represent the time series index to identify the time period information to which the index data belongs, that is, the time interval in which the index data is located.

[0064] Accordingly, in a possible embodiment, the attribute information corresponding to the index data includes at least one of a table identifier of the data table, a field identifier corresponding to the index data, and information about a time period to which the index data belongs.

[0065] Accordingly, determining the target index log file according to the hash value, the target log queue, and the attribute information corresponding to the index data includes:

[0066] Based on the hash value, a modulus operation is performed on the number of log queues corresponding to the target data table to obtain a modulus value corresponding to the index data. The implementation of this step has been described in detail above and will not be repeated here.

[0067] Afterwards, the log file corresponding to the queue identification information and the attribute information in the target log queue is determined as the target index log file, wherein the queue identification information includes the module value and the number of log queues corresponding to the data table.

[0068] Among them, the file name of the index log file can be determined in the above manner. When determining the index log file corresponding to the index data, the corresponding target index log file can be directly determined from the target log queue based on the queue identification information and attribute information. If the handle of the corresponding file is queried based on the above information, the index data can be written into the target index log file. If not found, the file name can be determined based on the above information, the target index log file is created and the index data is written into the target index log file, which can facilitate the subsequent query process of the generated index and improve the index query efficiency.

[0069] Therefore, through the above technical solution, the index data can be hashed, so as to provide data support for the balanced processing of the index data, ensure the load balance in the index data generation process, and improve the real-time performance of the index generation. At the same time, the binding of the index data and the target log queue can be realized, so that the information corresponding to the same index will be processed by the same log queue, so that in the subsequent query process based on the generated index, the index corresponding to the target log queue corresponding to the index data can be directly queried, effectively reducing the data access corresponding to the data query and improving the data query efficiency.

[0070] In a possible embodiment, the method further includes:

[0071] Determine the data writing duration corresponding to the target index log file.

[0072] By the above method, the file name of the target index log file to be written can be determined, and then the handle of the target index log file can be opened to write the index data into the target index log file. As an example, the suffix of the target index log file being written can be set to .irs_tmp, indicating that the file supports data writing.

[0073] In this embodiment, in order to ensure the real-time nature of index generation, each index log file can be set to record index data within a certain period of time. Based on this, in this embodiment, the data writing duration corresponding to the index log file can be obtained by counting from the time when the handle of the target index log file is opened.

[0074] Afterwards, if the data writing duration reaches a preset duration, the type of the target index log file is updated to a completion type, wherein the completion type file is a file to which no data is written any more.

[0075] The preset duration can be set according to the actual application scenario, such as the preset duration can be set shorter in scenarios with high index real-time requirements, and the preset duration can be set longer in scenarios with low index real-time requirements. It should be noted that after the type of the index log file is updated to the completion type, the index log file can be accessed by subsequent processing nodes for data playback to generate an index.

[0076] As an example, updating the type of the target index log file to the completion type may be to rename the suffix of the target index log file to .irs (index raw segment). This type of file does not support appending, so that no updated data will be appended to the index log file in the future, ensuring the accuracy and integrity of index data processing, avoiding data omissions during index generation based on the index log file, and ensuring the accuracy and effectiveness of index generation.

[0077] In actual application scenarios, index log files are usually written in a distributed manner, so the number of index log files is large but the data volume is small. If the index log files are directly dumped as indexes, the index files may be smaller, and the indexes may need to be merged later. Based on this, the present disclosure provides the following embodiments.

[0078] In a possible embodiment, the method may further include:

[0079] The multiple index log files under the log queue are grouped to obtain multiple file groups, wherein the target attribute information corresponding to the index log files in each file group is the same, and the target attribute information includes the queue identifier of the target log queue, the table identifier of the data table, and the field identifier of the index field corresponding to the index data. The table identifier can be the table name or the ID of the data table, and the field identifier can be the field name or the field ID, which can be preset, and the present disclosure does not limit this.

[0080] Among them, for each log queue, the file group can be determined by scanning the index log file. In this embodiment, the index log files with the same target attribute information are divided into the same group, so that the index log files in the same log queue where the index data of the same field in the same data table are located can be divided into the same group. As shown above, the index log files with the same (tableId, columnId, hashId / hashShard) can be divided into the same file group, wherein hashId / hashShard represents the queue identifier of the target log queue, indicating which queue among the multiple queues corresponding to the data table the target log queue is.

[0081] As another example, if the index log file name also includes timeBucket, the index log file can be further grouped by (tableId, columnId, hashId / hashShard, timeBucket) to obtain multiple file groups.

[0082] For each of the file groups, multiple index log files under the file group are merged to obtain a merged index log file. When subsequently generating an index for data playback, the index generation can be performed based on the merged index log file.

[0083] As an example, the contents of each index log file under the file group can be directly concatenated and merged to obtain a merged index log file, and the suffix of the merged index log file can be updated to .irm (index raw merge). When generating indexes later, the corresponding indexes can be generated for each file with the suffix irm, so that the index generated based on the index log files under the same group is comprehensive and complete, thereby improving the efficiency of data query based on the generated index and facilitating user query.

[0084] As another example, the size of the index log file will also affect the subsequent index generation efficiency, so a data volume threshold of the index log file can be set in advance. Accordingly, for each index log file under the file group, it can be merged from front to back according to the completion time of the index log file, while ensuring that the data volume of the merged index log file is less than the data volume threshold. For example, if the data volume after merging index log file X1 exceeds the data volume threshold, the index log files between X1 can be merged to obtain a merged index log file, and the merging continues from X1 to the back until the index log files under the file group are traversed and merged.

[0085] Through the above technical solution, the index log files can be further grouped, so that the index log files belonging to the same file group can be merged. On the one hand, the number of files that need to be indexed can be reduced and the efficiency of index generation can be improved. On the other hand, the subsequent secondary index merging process when the index is generated directly based on the index log file can be avoided, thereby ensuring the validity of the generated index and improving the efficiency of data query based on the index.

[0086] In a possible embodiment, the method may further include:

[0087] The amount of written data in each of the data tables during a target period of time is determined.

[0088] Among them, the target period can be set based on the actual application scenario, such as being set to the day closest to the current moment. The data volume can then be accumulated when the write data is written into the data table to determine the amount of data written in the data table within the target period.

[0089] The data volume is sent to the scheduler, so that the scheduler determines the log queue corresponding to each data table based on the received data volume corresponding to each data table.

[0090] Wherein, the scheduler receives the data volume written in the target time period by each data table, and then can dynamically adjust the quantity of the log queue corresponding to each data table according to the real-time data volume. For example, the log queue corresponding to each data table can be determined by the logic that each log queue processes the average data volume. If there are 8 log queues, i.e. Q0-Q7, in the initial case, log queues Q0-Q4 are allocated for data table T1, log queues Q5 and Q6 are allocated for data table T2, and log queues Q7 are allocated for data tables T3-T6.

[0091] In this embodiment, based on the amount of data written to data tables T1-T6 during the target period, the ratio of the data amounts of T1, T2, and T3-T6 is determined to be 4:3:1. The number of logs corresponding to data tables T1 and T2 can be further adjusted. For example, data table T1 can be set to allocate 4 log queues, data table T2 can be set to allocate 3 log queues, and data tables T3-T6 can be set to allocate 1 log queue.

[0092] For each of the data tables, the log queue corresponding to the data table sent by the scheduler is determined as a new log queue corresponding to the data table.

[0093] In response to receiving the message sent by the scheduler, the log queue corresponding to each data table can be determined according to the allocation of the log queue most recently sent by the scheduler, and then the target log queue to which the corresponding index data is written can be determined from the new log queue corresponding to the data table.

[0094] Therefore, through the above technical scheme, the dynamic allocation of log queues corresponding to data tables can be realized based on the amount of data written to each data table in the recent period, ensuring the matching degree of the data writing process of the log queue and the data table. At the same time, it can ensure to a certain extent that the amount of index data written in each log queue is roughly the same, ensuring load balancing in the index data processing process, avoiding delays in index data processing of some data tables due to uneven distribution of log queues, improving the real-time performance of index generation, and enhancing user experience.

[0095] The present disclosure also provides an index generation method, which may include:

[0096] Determine a to-be-processed log queue corresponding to an to-be-processed index log file, wherein the index log file is generated based on any one of the index data processing methods described above;

[0097] Obtaining the index log file from the to-be-processed log queue;

[0098] Among them, data in multiple log queues can be processed by multiple processing nodes, and the log queue corresponding to each processing node can be determined in a pre-configured manner. In this embodiment, the log queue to be processed corresponding to the current processing node can be determined based on the configuration information, and the corresponding index log file can be obtained from the log queue to be processed.

[0099] As an example, the scheduling of processing nodes and log queues can be realized through the scheduler. For example, the scheduler detects the processing node list through heartbeat or consul to determine the processing node that can consume the data in the log queue, and then allocates the same number of log queues to each processing node according to the total number of log queues, that is, determines the log queue to be processed corresponding to each processing node.

[0100] As an example, if the format of the index log file is irs, the various index log files can be merged based on the above-described merging method to obtain a merged index log file, such as a file in the irm format. Then, index generation processing is performed based on the merged index log file.

[0101] Data playback is performed based on the index data in the index log file to generate an index corresponding to the index data.

[0102] As an example, in this step, index generation can be performed through data playback, wherein the index generation can be constructed using a structure such as a B+ tree, which can be constructed using a common construction method in the art and will not be described in detail here.

[0103] Therefore, through the above technical solution, the index log files to be processed can be obtained from their corresponding log queues to be processed, so that the load balancing in the index generation process can be guaranteed to a certain extent through the scheduling of the log queue, the efficiency of index generation can be improved, the delay of user data query can be reduced, and the user experience can be improved.

[0104] In the actual operation of the system, there may be several situations that cause the originally balanced load between nodes to suddenly become unbalanced, such as the index traffic corresponding to some data tables suddenly increases, or the computing power of the processing nodes corresponding to the log queues to be processed is not exactly the same, resulting in abnormally high loads on some processing nodes; or a processing node is abnormal, causing the log queue it consumes to be interrupted for a period of time, resulting in a delay. Based on this, the present disclosure also provides the following embodiments. Specifically, the method may also include:

[0105] When the data volume corresponding to the to-be-processed log queue exceeds the processing threshold, the data volumes corresponding to other log queues except the to-be-processed log queue are obtained.

[0106] Among them, the processing threshold can be set based on the processing performance of the processing node corresponding to the log queue to be processed. In the present disclosure, if the amount of data corresponding to the log queue to be processed exceeds the processing threshold, it means that there is a risk of overload in the processing of the processing node corresponding to the log queue to be processed. At this time, the amount of data corresponding to other log queues can be obtained to determine the load conditions of other log queues.

[0107] The scheduling log queue is determined according to the data volume corresponding to the to-be-processed log queue and the data volume corresponding to other log queues.

[0108] As an example, the system is configured with an idle queue, then when it is determined that the data volume of other log queues is 0, the other log queues can be directly determined as the scheduling log queues. As another example, other log queues with the minimum data volume can be obtained, if the data volume corresponding to the other log queues is less than the data volume corresponding to the pending log queues, the other log queues can be determined as the scheduling log queues. As another example, the processing node corresponding to each log queue can determine its corresponding processing load based on its data volume and self-processing performance, then in this embodiment, other log queues with the minimum corresponding processing load can be obtained, if the processing load corresponding to the other log queues is less than the processing load corresponding to the pending log queues, the other log queues can be determined as the scheduling log queues.

[0109] At least a portion of the index log files in the to-be-processed log queue is moved to the scheduling log queue, that is, the portion of the index log files is stored in the scheduling log queue for processing.

[0110] As an example, the index log file corresponding to the amount of data exceeding the processing threshold in the pending log queue can be moved to the scheduling log queue for processing. As another example, it is also possible to average the amount of data corresponding to the pending log queue and the amount of data corresponding to the scheduling log queue, thereby determining the index log file in the pending log queue that needs to be moved to the scheduling log queue.

[0111] Therefore, through the above technical solution, the scheduling of index log files in multiple log queues can be realized. When the index amount of the log queue is unbalanced or the data amount of a log queue is overloaded due to some external reasons, at least a part of the index log files in the overloaded log queue is moved to other idle queues, so as to quickly eliminate the delay, thereby solving the write delay caused by the imbalance of node load in the distributed index generation process and further improving the efficiency of index generation.

[0112] The present disclosure also provides a device for processing index data, such as Figure 2 As shown, the device 10 comprises:

[0113] A first acquisition module 100 is used to acquire index data corresponding to target data written in the data table when writing data in the data table, wherein the index data is data corresponding to an index field in the target data used to construct an index;

[0114] The first determination module 200 is used to determine the target log queue and the target index log file to be written into each of the index data based on the log queue corresponding to the data table, wherein each of the log queues stores a plurality of index log files, and each of the index log files stores a plurality of index data;

[0115] The first processing module 300 is used to write the index data corresponding to the target data into the target index log file in the target log queue, wherein the index log file is a pre-written log file, and the index log file is used for data playback to generate an index corresponding to the index data.

[0116] Optionally, the first determining module includes:

[0117] A first processing submodule, configured to perform a hash operation on the index data to obtain a hash value of the index data;

[0118] A first determination submodule, configured to determine the target log queue according to the hash value and the log queue corresponding to the data table;

[0119] The second determination submodule is used to determine the target index log file according to the hash value, the target log queue and the attribute information corresponding to the index data.

[0120] Optionally, the attribute information corresponding to the index data includes at least one of a table identifier of the data table, a field identifier of an index field corresponding to the index data, and information about a time period to which the index data belongs;

[0121] The second determining submodule includes:

[0122] A second processing submodule is used to perform a modulo operation on the number of log queues corresponding to the target data table based on the hash value to obtain a modulo value corresponding to the index data;

[0123] The third determination submodule is used to determine the log file corresponding to the queue identification information and the attribute information in the target log queue as the target index log file, wherein the queue identification information includes the module value and the number of log queues corresponding to the data table.

[0124] Optionally, the device further comprises:

[0125] A second determination module is used to determine the data writing duration corresponding to the target index log file;

[0126] The first updating module is used to update the type of the target index log file to a completion type if the data writing duration reaches a preset duration, wherein the completion type file is a file to which no data is written any more.

[0127] Optionally, the device further comprises:

[0128] A second processing module is used to group the multiple index log files under the log queue to obtain multiple file groups, wherein the target attribute information corresponding to the index log files in each file group is the same, and the target attribute information includes a queue identifier of the target log queue, a table identifier of the data table, and a field identifier of an index field corresponding to the index data;

[0129] The merging module is used to merge the multiple index log files under each file group to obtain a merged index log file.

[0130] Optionally, the device further comprises:

[0131] A third determining module, used to determine the amount of data written into each of the data tables in a target period of time;

[0132] A sending module, used for sending the data volume to the scheduler, so that the scheduler determines the log queue corresponding to each data table based on the received data volume corresponding to each data table;

[0133] The second updating module is used to determine, for each of the data tables, the log queue corresponding to the data table sent by the scheduler as a new log queue corresponding to the data table.

[0134] The present disclosure also provides an index generating device, the device comprising:

[0135] A fourth determination module is used to determine a to-be-processed log queue corresponding to an to-be-processed index log file, wherein the index log file is generated based on the index data processing method described above;

[0136] A second acquisition module is used to acquire the index log file from the to-be-processed log queue;

[0137] A generation module is used to perform data playback based on the index data in the index log file and generate an index corresponding to the index data.

[0138] Optionally, the device further comprises:

[0139] A third acquisition module is used to acquire the data volume corresponding to other log queues except the log queue to be processed when the data volume corresponding to the log queue to be processed exceeds the processing threshold;

[0140] A fifth determination module, configured to determine a scheduling log queue according to the data volume corresponding to the to-be-processed log queue and the data volume corresponding to other log queues;

[0141] The third processing module is used to move at least a part of the index log files in the to-be-processed log queue to the scheduled log queue.

[0142] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device (such as a terminal device or a server) 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0143] like Figure 3 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0144] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0145] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0146] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0147] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0148] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0149] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains index data corresponding to the write data in the data table; determines the target log queue and index log file corresponding to each index data based on the log queue corresponding to the data table; writes the index data into the index log file in the target log queue, wherein the index log file is a pre-written log file, and the index log file is used for data playback to generate an index corresponding to the index data.

[0150] Alternatively, the computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: determines a log queue to be processed corresponding to an index log file to be processed, wherein the index log file is generated based on the index data processing method described above; obtains the index log file from the log queue to be processed; and performs data playback based on the index data in the index log file to generate an index corresponding to the index data.

[0151] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the first acquisition module may also be described as "a module for acquiring index data corresponding to the target data written in the data table when writing data in the data table".

[0154] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0155] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0156] According to one or more embodiments of the present disclosure, Example 1 provides a method for processing index data, wherein the method includes:

[0157] When writing data in a data table, obtaining index data corresponding to the target data written in the data table, wherein the index data is data corresponding to an index field in the target data used to construct an index;

[0158] Based on the log queue corresponding to the data table, determine the target log queue and the target index log file to which each of the index data is to be written, wherein each of the log queues stores a plurality of index log files, and each of the index log files stores a plurality of index data;

[0159] The index data corresponding to the target data is written into the target index log file in the target log queue, wherein the index log file is a write-ahead log file used for data playback to generate an index corresponding to the index data.

[0160] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein determining the target log queue and the target index log file to be written to each index data based on the log queue corresponding to the data table includes:

[0161] Performing a hash operation on the index data to obtain a hash value of the index data;

[0162] Determine the target log queue according to the hash value and the log queue corresponding to the data table;

[0163] The target index log file is determined according to the hash value, the target log queue, and the attribute information corresponding to the index data.

[0164] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, wherein the attribute information corresponding to the index data includes at least one of a table identifier of the data table, a field identifier of an index field corresponding to the index data, and information about a time period to which the index data belongs;

[0165] The step of determining the target index log file according to the hash value, the target log queue, and the attribute information corresponding to the index data includes:

[0166] Performing a modulo operation on the number of log queues corresponding to the target data table based on the hash value to obtain a modulo value corresponding to the index data;

[0167] The log file corresponding to the queue identification information and the attribute information in the target log queue is determined as the target index log file, wherein the queue identification information includes the module value and the number of log queues corresponding to the data table.

[0168] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 1, wherein the method further includes:

[0169] Determine the data writing duration corresponding to the target index log file;

[0170] If the data writing duration reaches a preset duration, the type of the target index log file is updated to a completion type, wherein the completion type file is a file to which no data is written any more.

[0171] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 1, wherein the method further includes:

[0172] Grouping the multiple index log files under the log queue to obtain multiple file groups, wherein the target attribute information corresponding to the index log files in each file group is the same, and the target attribute information includes a queue identifier of the target log queue, a table identifier of the data table, and a field identifier of an index field corresponding to the index data;

[0173] For each of the file groups, multiple index log files under the file group are merged to obtain a merged index log file.

[0174] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 1, wherein the method further includes:

[0175] Determine the amount of data written to each of the data tables during a target period of time;

[0176] Sending the data volume to the scheduler, so that the scheduler determines the log queue corresponding to each data table based on the received data volume corresponding to each data table;

[0177] For each of the data tables, the log queue corresponding to the data table sent by the scheduler is determined as a new log queue corresponding to the data table.

[0178] According to one or more embodiments of the present disclosure, Example 7 provides an index generation method, the method comprising:

[0179] Determine a to-be-processed log queue corresponding to an to-be-processed index log file, wherein the index log file is generated based on the index data processing method described in any one of Examples 1-6;

[0180] Obtaining the index log file from the to-be-processed log queue;

[0181] Data playback is performed based on the index data in the index log file to generate an index corresponding to the index data.

[0182] According to one or more embodiments of the present disclosure, Example 8 provides the method of Example 7, wherein the method further includes:

[0183] When the data volume corresponding to the to-be-processed log queue exceeds the processing threshold, obtaining the data volume corresponding to other log queues except the to-be-processed log queue;

[0184] Determine the scheduling log queue according to the data volume corresponding to the pending log queue and the data volume corresponding to other log queues;

[0185] At least a portion of the index log files in the to-be-processed log queue is moved to the scheduled log queue.

[0186] According to one or more embodiments of the present disclosure, Example 9 provides a device for processing index data, the device comprising:

[0187] A first acquisition module is used to acquire index data corresponding to target data written in the data table when writing data in the data table, wherein the index data is data corresponding to an index field in the target data used to construct an index;

[0188] A first determination module is used to determine a target log queue and a target index log file to be written into each of the index data based on the log queue corresponding to the data table, wherein each of the log queues stores a plurality of index log files, and each of the index log files stores a plurality of index data;

[0189] The first processing module is used to write the index data corresponding to the target data into the target index log file in the target log queue, wherein the index log file is a pre-written log file, and the index log file is used for data playback to generate an index corresponding to the index data.

[0190] According to one or more embodiments of the present disclosure, Example 10 provides an index generating device, the device comprising:

[0191] A fourth determination module, used to determine a to-be-processed log queue corresponding to an to-be-processed index log file, wherein the index log file is generated based on the index data processing method described in any one of Examples 1-6;

[0192] A second acquisition module is used to acquire the index log file from the to-be-processed log queue;

[0193] A generation module is used to perform data playback based on the index data in the index log file and generate an index corresponding to the index data.

[0194] According to one or more embodiments of the present disclosure, Example 11 provides a computer-readable medium having a computer program stored thereon, which implements the steps of any of the methods described in Examples 1-8 when executed by a processing device.

[0195] According to one or more embodiments of the present disclosure, Example 12 provides an electronic device, including:

[0196] a storage device having a computer program stored thereon;

[0197] A processing device is used to execute the computer program in the storage device to implement the steps of any one of the methods described in Examples 1-8.

[0198] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0199] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0200] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A method for processing index data, characterized in that: The method comprises: When writing data in a data table, obtaining index data corresponding to the target data written in the data table, wherein the index data is data corresponding to an index field in the target data used to construct an index; Based on the log queue corresponding to the data table, determine the target log queue and the target index log file to which each of the index data is to be written, wherein each of the log queues stores a plurality of index log files, and each of the index log files stores a plurality of index data; The index data corresponding to the target data is written into the target index log file in the target log queue, wherein the index log file is a write-ahead log file used for data playback to generate an index corresponding to the index data.

2. The method according to claim 1, characterized in that The step of determining a target log queue and a target index log file to be written to each index data based on the log queue corresponding to the data table includes: Performing a hash operation on the index data to obtain a hash value of the index data; Determine the target log queue according to the hash value and the log queue corresponding to the data table; The target index log file is determined according to the hash value, the target log queue, and the attribute information corresponding to the index data.

3. The method according to claim 2, characterized in that The attribute information corresponding to the index data includes at least one of a table identifier of the data table, a field identifier of an index field corresponding to the index data, and information about a time period to which the index data belongs; The step of determining the target index log file according to the hash value, the target log queue, and the attribute information corresponding to the index data includes: Perform a modulo operation on the number of log queues corresponding to the target data table based on the hash value to obtain a modulo value corresponding to the index data; The log file corresponding to the queue identification information and the attribute information in the target log queue is determined as the target index log file, wherein the queue identification information includes the module value and the number of log queues corresponding to the data table.

4. The method according to claim 1, characterized in that The method further comprises: Determine the data writing duration corresponding to the target index log file; If the data writing duration reaches a preset duration, the type of the target index log file is updated to a completion type, wherein the completion type file is a file to which no data is written any more.

5. The method according to claim 1, characterized in that The method further comprises: Grouping the multiple index log files under the log queue to obtain multiple file groups, wherein the target attribute information corresponding to the index log files in each file group is the same, and the target attribute information includes a queue identifier of the target log queue, a table identifier of the data table, and a field identifier of an index field corresponding to the index data; For each of the file groups, multiple index log files under the file group are merged to obtain a merged index log file.

6. The method according to claim 1, characterized in that The method further comprises: Determine the amount of data written to each of the data tables during a target period of time; Sending the data volume to the scheduler, so that the scheduler determines the log queue corresponding to each data table based on the received data volume corresponding to each data table; For each of the data tables, the log queue corresponding to the data table sent by the scheduler is determined as a new log queue corresponding to the data table.

7. An index generation method, characterized in that: The method comprises: Determine a to-be-processed log queue corresponding to an to-be-processed index log file, wherein the index log file is generated based on the index data processing method according to any one of claims 1 to 6; Obtaining the index log file from the to-be-processed log queue; Data playback is performed based on the index data in the index log file to generate an index corresponding to the index data.

8. The method according to claim 7, characterized in that The method further comprises: When the data volume corresponding to the to-be-processed log queue exceeds the processing threshold, obtaining the data volume corresponding to other log queues except the to-be-processed log queue; Determine the scheduling log queue according to the data volume corresponding to the pending log queue and the data volume corresponding to other log queues; At least a portion of the index log files in the to-be-processed log queue is moved to the scheduled log queue.

9. A device for processing index data, characterized in that: The device comprises: A first acquisition module is used to acquire index data corresponding to target data written in the data table when writing data in the data table, wherein the index data is data corresponding to an index field in the target data used to construct an index; A first determination module is used to determine a target log queue and a target index log file to be written into each of the index data based on the log queue corresponding to the data table, wherein each of the log queues stores a plurality of index log files, and each of the index log files stores a plurality of index data; The first processing module is used to write the index data corresponding to the target data into the target index log file in the target log queue, wherein the index log file is a pre-written log file, and the index log file is used for data playback to generate an index corresponding to the index data.

10. An index generating device, characterized in that: The device comprises: a fourth determining module, configured to determine a to-be-processed log queue corresponding to an to-be-processed index log file, wherein the index log file is generated based on the index data processing method according to any one of claims 1 to 6; A second acquisition module is used to acquire the index log file from the to-be-processed log queue; A generation module is used to perform data playback based on the index data in the index log file and generate an index corresponding to the index data.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 8 are implemented.

12. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 8.

Images