Index Reconstruction Method, Apparatus, Device, and Medium

Abstract

The present disclosure provides an index rebuilding method applied to a database system, which can be used in the financial field. Among them, the database system includes a data service and an index service, and the data service and the index service are respectively used to store and manage the original data and index data of the database system. The method includes: when there is a change in the index data structure of the stock data, reading the stock data from the data service; and rebuilding the index of the stock data in the index service according to the changed index data structure; wherein, during the process of rebuilding the index of the stock data, the rebuilding progress of the index of the stock data is updated and controlled in real time. The present disclosure also provides an index rebuilding device, equipment, storage medium and program product applied to the database system.

Description

Technical Field

[0001] The present disclosure relates to the field of big data, and more particularly to an index reconstruction method, apparatus, device, medium and program product applied to a database system. Background Art

[0002] An index is metadata stored by a database system for conditional queries (WHERE statements), which enables the database server to find and retrieve specific data records much faster than without an index. In related technologies, the index and data are usually stored together, i.e., on the same physical machine. When the data structure of the index is changed by an operation using the Database Schema Definition Language (DDL), index reconstruction is triggered. Since the index and data are stored together, resource usage and start / stop of the reconstruction task cannot be controlled during index reconstruction, which can cause a significant impact on the performance of the database system during index reconstruction. Summary of the Invention

[0003] In view of the above problems, the present disclosure provides an index reconstruction method, apparatus, device, medium and program product applied to a database system that can perform real-time update and control of the index reconstruction task.

[0004] In a first aspect of an embodiment of the present disclosure, there is provided an index reconstruction method applied to a database system, where the database system includes a data service and an index service, and the data service and the index service are respectively used to store and manage the original data and index data of the database system. The method includes: when there is a change in the index data structure related to the stock data, reading the stock data from the data service; and reconstructing the index of the stock data in the index service according to the changed index data structure; wherein, during the process of reconstructing the index of the stock data, the reconstruction progress of the index of the stock data is updated and controlled in real time.

[0005] According to an embodiment of the present disclosure, the reading of the stock data from the data service further includes: reading the stock data through at least one scan, wherein during each scan, data is quantitatively read according to the amount of data applied for this scan.

[0006] According to an embodiment of the present disclosure, the reading of the stock data from the data service further includes: applying for the amount of data for this scan before each scan.

[0007] According to an embodiment of the present disclosure, the rebuilding of the indexes of the inventory data in the index service according to the changed index data structure further includes: before rebuilding the index corresponding to any primary key in the index service according to the changed index data structure, sending a hold lock for the index corresponding to the primary key to the index service; wherein, the hold lock is used to block a request from the data service to the index service to write data into the index corresponding to the primary key; and after the index corresponding to the primary key is rebuilt, releasing the hold lock.

[0008] According to an embodiment of the present disclosure, wherein, the real-time updating of the rebuilding progress of the indexes of the inventory data includes: after the index corresponding to any primary key is rebuilt, updating the rebuilding progress to the primary key name of the primary key.

[0009] According to an embodiment of the present disclosure, the controlling of the rebuilding progress of the indexes of the inventory data includes: during the process of rebuilding the indexes of the inventory data, when receiving an instruction to pause or resume the rebuilding of the indexes of the inventory data, pausing or resuming the rebuilding of the indexes of the inventory data.

[0010] According to an embodiment of the present disclosure, the resuming of the rebuilding of the indexes of the inventory data includes: starting to resume the rebuilding from the index corresponding to the primary key after the primary key shown in the rebuilding progress.

[0011] On the other hand, an embodiment of the present disclosure provides an index rebuilding apparatus applied to a database system. The database system includes a data service and an index service, and the data service and the index service are respectively used to store and manage the original data and index data of the database system. The apparatus includes a scanning module, a rebuilding module, and a control module. The scanning module includes at least one scanning thread, and is used to read the inventory data from the data service through the at least one scanning thread when there is a change in the index data structure of the inventory data.

[0012] The rebuilding module includes at least one rebuilding thread, and is used to rebuild the indexes of the inventory data in the index service through the at least one rebuilding thread according to the changed index data structure. The control module includes a control thread, and is used to update and control the rebuilding progress of the indexes of the inventory data in real time through the control thread during the process of rebuilding the indexes of the inventory data.

[0013] According to an embodiment of the present disclosure, the apparatus further includes a resource limiter. The resource limiter is used to set the currently allowed amount of scanned data according to the configuration of the control thread. Wherein, before each scanning thread scans the data service, it applies to the resource limiter for the amount of data to be scanned this time, and each scanning thread scans the data service once according to the applied amount of data to be scanned this time.

[0014] According to an embodiment of the present disclosure, the reconstruction module is further configured to, before each reconstruction thread reconstructs the index corresponding to any primary key in the index service, initiate a hold lock on the index corresponding to the primary key to the index service through the reconstruction thread; wherein the hold lock is used to block a request from the data service to write data into the index corresponding to the primary key to the index service; and after the index corresponding to the primary key is reconstructed, the reconstruction thread releases the hold lock.

[0015] According to an embodiment of the present disclosure, the control module is further configured to, after the index corresponding to any primary key is reconstructed, update the reconstruction progress to the primary key name of the primary key through the control thread.

[0016] According to an embodiment of the present disclosure, the control module is further configured to, during the process of reconstructing the index of the inventory data, when receiving an instruction to pause or resume the reconstruction of the index of the inventory data, pause or resume the reconstruction of the index of the inventory data through the control thread.

[0017] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes one or more processors and one or more memories. The one or more memories are used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the above-mentioned index reconstruction method.

[0018] In a fourth aspect of the present disclosure, a computer-readable storage medium is further provided, on which executable instructions are stored, and when the instructions are executed by a processor, the processor is caused to execute the above-mentioned index reconstruction method.

[0019] In a fifth aspect of the present disclosure, a computer program product is further provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned index reconstruction method is implemented. Description of the Drawings

[0020] Through the following description of the embodiments of the present disclosure with reference to the drawings, the above-mentioned content and other objects, features, and advantages of the present disclosure will become clearer. In the drawings:

[0021] Figure 1 Schematically shows the system architecture of an index reconstruction method, apparatus, device, medium, and program product according to an embodiment of the present disclosure;

[0022] Figure 2 Schematically shows a schematic diagram of the data structure of the original data stored in the data service and the index data stored in the index service according to an embodiment of the present disclosure;

[0023] Figure 3Schematically shows a flowchart of an index reconstruction method according to an embodiment of the present disclosure;

[0024] Figure 4 Schematically shows an operation process of reconstructing an index of stock data in the index reconstruction method according to an embodiment of the present disclosure;

[0025] Figure 5 Schematically shows an operation process of real-time updating and controlling the index reconstruction progress in the index reconstruction method according to an embodiment of the present disclosure;

[0026] Figure 6 Schematically shows a framework diagram of an index reconstruction service according to an embodiment of the present disclosure;

[0027] Figure 7 Schematically shows a block diagram of an index reconstruction apparatus according to an embodiment of the present disclosure; and

[0028] Figure 8 Schematically shows a block diagram of an electronic device suitable for implementing the index reconstruction method according to an embodiment of the present disclosure. Detailed embodiments

[0029] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure. However, obviously, one or more embodiments may be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present disclosure.

[0030] The terms used herein are merely for describing specific embodiments and are not intended to limit the present disclosure. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0031] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0032] In the case of using expressions such as "at least one of A, B, and C", generally, it should be interpreted according to the meaning that those skilled in the art usually understand this expression (for example, "a system having at least one of A, B, and C" should include, but is not limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0033] To solve the problem that the index rebuilding task of the database system in the related art is uncontrollable, the embodiments of the present disclosure provide an index rebuilding method, device, equipment, medium, and program product applied to the database system.

[0034] Among them, in the database system of the embodiments of the present disclosure, the data service and the index service are respectively used to store and manage the original data and index data of the database system. In this way, the index service and the data service can be decoupled, and the index data can be separated from the original data of the database system.

[0035] Furthermore, when an index data structure change of the stock data requires index rebuilding, according to the embodiments of the present disclosure, the stock data can be read from the data service, and the index of the stock data can be rebuilt in the index service according to the changed index data structure. Among them, the rebuilding progress is updated and controlled in real time during the index rebuilding task. The embodiments of the present disclosure perform index rebuilding through the interaction with the index service and the data service, so that the index rebuilding task can be independent of the data service and the index service, which greatly facilitates the independent control of the index rebuilding task and can achieve the overall controllability of the index rebuilding task.

[0036] It should be noted that in this article, "original data" is used to indicate the data stored in the data table in the database system, so as to distinguish it from "index data". And "stock data" refers to the data that has been stored in the data table of the database system before the change of the data structure of the index data, that is, the "stock data" in this article belongs to a part of the "original data".

[0037] Figure 1 Schematically shows a system architecture 100 of an index rebuilding method, device, equipment, medium, and program product according to an embodiment of the present disclosure.

[0038] As Figure 1 shown, in the system architecture 100, the operation process in the index rebuilding task can be service-ized to form an index rebuilding service 101. When an index data structure change of the stock data requires index rebuilding, the index rebuilding task is executed through the interaction of the index rebuilding service 101, the data service 102, and the index service 103.

[0039] The data service 102 and the index service 103 are respectively used to store and manage the original database data and the index data. The index data stored in the index service 103 is the metadata of the original data stored in the data service 102, specifically the metadata stored for conditional queries (where statements), which is essentially the mapping relationship from the values of user data to keys. An index is a separate, physical storage structure that sorts the values of one or more fields in a database table, consisting of a set of values of one or several fields in the original data and a list of logical pointers corresponding to the data pages in the table that physically identify these values. The role of an index is equivalent to the table of contents of a book, and the required content can be quickly found according to the page numbers in the table of contents. The index provides pointers to the data values stored in the specified fields in the database, and then sorts these pointers according to the specified sorting order. The database uses the index to find a specific value, and then follows the pointer to find the complete record containing that value. This can make the SQL statements for the corresponding data table execute faster and can quickly access specific information in the database table.

[0040] Figure 2 Schematically shows a schematic diagram of the data structure of the original data stored in the data service 102 and the index data stored in the index service 103 according to an embodiment of the present disclosure.

[0041] As Figure 2 shown, the original data is stored in the underlying storage engine in the form of key-value pairs, and the logical record of each record can be schematically represented as: (primary key, (field 1, field 1 content), (field 2, field 2 content),......). When the original data is a row-stored data table, this record corresponds to a row in the data table.

[0042] Correspondingly, the index data can also be stored in the underlying storage engine in the form of ((field 1, field 1 content), primary key). Of course, Figure 2 that only one field (i.e., field 1) is involved in the index is only exemplary. The index data can be a single-field index or a multi-field index, which is not limited in the present disclosure.

[0043] When performing a data query through an index condition, combining Figure 1 and Figure 2 , a range query of the index can be first performed in the index service 103 according to the index condition (for example, the value or value range of field 1 given in the where conditional statement) to find the index data that meets the index condition, and then the primary key stored in the found index data is returned, and then the corresponding data can be obtained in the data service 102 according to the primary key.

[0044] When the amount of raw data in the data service 102 continuously grows or changes, the corresponding index data in the index service 103 also continuously grows. After there is existing data in the raw data of the data service 102, if a user changes the index in the schema, it is necessary to reconstruct the index for the existing data; otherwise, the index of the existing data will no longer be usable. The schema is the organization and structure of a database, including schema objects, which can be tables, columns, data types, views, stored procedures, relationships, primary keys, foreign keys, etc. If the upstream service of the database system triggers a schema change, and this schema change operation involves changing the index of the existing data in the raw data, it is necessary to change the data structure of the index of this existing data in the index service 103. For example, in the index data structure shown in Figure 2 add or delete fields in the index, or replace fields in the index data, or change the names of fields in the index data, etc.

[0045] According to an embodiment of the present disclosure, in the system architecture 100, an index reconstruction service 101 is introduced to execute the index reconstruction task. Specifically, when there is a change in the index data structure involving existing data, the index reconstruction service 101 can read the existing data from the data service 102 by scanning, and then reconstruct the index of the existing data in the index service 103 according to the changed index data structure. Among them, by service-ifying the index reconstruction process through the index reconstruction service 101, the reconstruction progress of the index of the existing data can be updated and controlled in real time through the configuration in the index reconstruction service 101. In this way, the overall controllability of the index reconstruction task is achieved.

[0046] The following will combine Figure 1 and Figure 2 system architecture and data structure examples, and introduce in detail the index reconstruction method applied to the database system according to the embodiments of the present disclosure through Figures 3 to 6 . Among them, this method can be executed by the index reconstruction service 101.

[0047] Figure 3 Schematically shows a flowchart of the index reconstruction method according to an embodiment of the present disclosure.

[0048] As Figure 3 shown, the index reconstruction method according to this embodiment may include operation S310 to operation S330.

[0049] First, in operation S310, when there is a change in the index data structure of the stock data, the stock data is read from the data service 102. For example, when an upstream business party of the database system triggers a schema change, it is detected whether this schema change operation involves an index change of the stock data. If it does not involve an index change of the stock data, for example, no index change is included or a new data table is involved, there is no need to perform index reconstruction. If it involves an index change, for example, adding or modifying an index in an existing data table, it belongs to the situation of a change in the index data structure of the stock data.

[0050] When reading the stock data from the data service 102, the stock data can be read through at least one scan. For example, when the amount of stock data involved is relatively large, the stock data can be read by multiple scan threads scanning in parallel.

[0051] According to an embodiment of the present disclosure, the data volume for each scan can be pre-applied, and the data is read quantitatively according to the applied data volume for this scan. For example, in the index reconstruction service 101, the maximum data volume that the index reconstruction service 101 is allowed to read from the data service 102 at each moment can be limited. Then, before each scan, for example, each scan thread first applies for the data volume for this scan from the above maximum data volume. In this way, the resources consumed by the index reconstruction operation can be controlled and speed-limited. In addition, the index reconstruction service 101 can be allowed to configure or adjust the above maximum data volume according to the resource usage situation of the cluster, so that the resource usage of the index reconstruction task can be limited and adjustable.

[0052] Then, in operation S320, the index of the stock data is reconstructed in the index service 103 according to the changed index data structure. Combining Figure 2 , assuming that the changed index data structure is ((field 2, field 2 content), primary key), then when reconstructing the index of the stock data, the index reconstruction service 101 can determine the index data range to be modified according to the primary key of the stock data, and replace the field data in the index corresponding to these primary keys, or delete the original index corresponding to these primary keys, and rewrite the changed index in the index service 103.

[0053] Meanwhile, in operation S330, during the process of reconstructing the index of the stock data, the reconstruction progress of the index of the stock data is updated and controlled in real time. For example, after the index reconstruction corresponding to any primary key is completed, the reconstruction progress can be updated to the primary key name of the primary key, so as to update the index reconstruction progress in real time. Furthermore, the index reconstruction task can be started and stopped in real time. For example, it can be paused or resumed according to business needs or resource consumption, so that the overall resource consumption of the index reconstruction task is controllable, and an uncontrollable impact on the performance of the database system is avoided.

[0054] It can be seen that in the embodiment of the present disclosure, the index reconstruction service 101 reconstructs the index through interaction with the index service 103 and the data service 102, enabling the index reconstruction task to be independent of the data service 102 and the index service 103. This greatly facilitates the separate control of the index reconstruction task and can achieve the overall controllability of the index reconstruction task.

[0055] Figure 4 Schematically shows the operation process of reconstructing the index of the stock data in operation S320 in the index reconstruction method according to the embodiment of the present disclosure.

[0056] As Figure 4 shown, according to the embodiment of the present disclosure, reconstructing the index of the stock data in operation S320 may include operation S401 and operation S402. The following is described in conjunction with Figure 1 this.

[0057] In operation S401, before reconstructing the index corresponding to any primary key in the index service 103 according to the changed index data structure, a hold lock 11 for the index corresponding to the primary key is initiated to the index service 103. The hold lock 11 is used to block the data service 102 from initiating a request to write data into the index corresponding to the primary key to the index service 103.

[0058] In operation S402, after the index corresponding to the primary key is reconstructed, the hold lock 11 is released.

[0059] Specifically, while the index reconstruction service 101 reconstructs the index of the stock data, the data in the data service 102 may continuously change with the upstream business, and accordingly, the index corresponding to the corresponding primary key in the index service 103 needs to be changed. This causes the data service 102 to continuously initiate real-time write requests (S1) to the index service 103. For example, when the changed index data structure is ((field 2, field 2 content), primary key), if the value in field 2 of a certain stock data changes, the data service 102 will initiate a real-time write request to the index service 103 to update the content of field 2 in the index of the data.

[0060] However, if the index reconstruction of the stock data is not completed before the real-time write request reaches the index service 103, chaos will occur. How to ensure that the data is consistent with the original data after the index reconstruction request (S2) and the real-time write request (S1) process the index corresponding to the same primary key is crucial for the success of the index reconstruction.

[0061] In response to this, the embodiments of the present disclosure provide a holding lock 11 to block the data service 102 from initiating a real-time write request to the index service 103 before the index reconstruction is completed. That is, when the index reconstruction service 101 is reconstructing the index corresponding to a certain primary key, it temporarily blocks the request of the data service 102 to write data to the index corresponding to the primary key, and waits until the index reconstruction corresponding to the primary key is successful, and then releases the holding lock 11. Then, the data service 101 is allowed to update the index corresponding to the primary key. In this way, the consistency between the index data in the index service 103 and the original data in the data service 102 can be ensured.

[0062] The granularity of this holding lock is the index corresponding to the primary key for which the index is currently being reconstructed, and the granularity is relatively small. For example, when the original data in the data service 102 is row-store data, the holding lock is a row-level lock, which can be used to block only the real-time write request of the data service 102 to the index of the row data being reconstructed, and will release the real-time write request after the index reconstruction of the row is completed. It can be seen that the blocking time of this holding lock is relatively short, and while ensuring data consistency, the impact on the data service 102 and the index service 103 is controllable.

[0063] Figure 5 Schematically shows the operation process of real-time updating and controlling the index reconstruction progress in the index reconstruction method according to the embodiments of the present disclosure.

[0064] As Figure 5 shown, the process of real-time updating and controlling the index reconstruction progress in operation S330 according to the embodiments of the present disclosure may include operation S501 to operation S503.

[0065] In operation S501, after the index reconstruction corresponding to any primary key is completed, the reconstruction progress is updated to the primary key name of the primary key.

[0066] In operation S502, when receiving an instruction to pause the reconstruction of the index of the stock data, the reconstruction of the index of the stock data is paused.

[0067] In operation S503, when receiving an instruction to resume the reconstruction of the index of the stock data, the reconstruction is resumed starting from the index corresponding to the primary key after the primary key shown in the reconstruction progress.

[0068] In this way, in the index reconstruction method of the embodiments of the present disclosure, the index reconstruction task can be interrupted and resumed at any time. For example, the index reconstruction process can be started and paused at any time according to the cluster load situation. Thus, the overall index reconstruction task is controllable.

[0069] Figure 6 Schematically shows the framework diagram of the index reconstruction service 101 according to the embodiments of the present disclosure. It should be noted that Figure 6The framework shown is an embodiment of the index reconstruction service 101 and does not limit the present disclosure.

[0070] As Figure 6 shown, the index reconstruction service 101 may include a meta-information storage 601, a control thread 602, multiple parallel scanning threads 603, multiple parallel reconstruction threads 604, and a resource throttler 605.

[0071] When the upstream business party triggers a schema change involving a change in the index data structure of the existing data, the index reconstruction service 101 first preprocesses and persists the meta-information related to the index reconstruction task to a third-party storage system (such as ZooKeeper or etcd) through the meta-information storage 601. Among them, the relevant meta-information is shown in Table 1 below, mainly including task description, original content of the index item, updated content of the index item, index reconstruction progress information, etc.

[0072] Next, the index reconstruction service 101 performs the index reconstruction operation through the mutual cooperation of the control thread, a group of scanning threads, a throttler, and the reconstruction threads.

[0073] Specifically, the main tasks of the control thread 602 are: dividing the data to be reconstructed into intervals, generating scanning tasks and starting the corresponding scanning threads 603, starting the reconstruction threads 604, and being responsible for progress storage and interaction with the outside, such as pausing or resuming the reconstruction task, etc.

[0074]

[0075] Table 1

[0076] The scanning thread 603 performs a primary key scan (scan) on the data service 102 according to the allocated scanning interval, reads the fields required for this reconstruction task, encodes the data according to the changed index data structure to form index data in key-value format, forms an index data packet after encoding, and sends it to the task queue of the reconstruction thread 604. When performing a primary key scan each time, the scanning thread 603 can apply to the resource throttler 605 for the amount of data to be scanned this time. The amount of data to be scanned this time can be, for example, the number of bytes of the fields read, and in this way, the use of disk resources can be controlled.

[0077] The reconstruction thread 604 can, according to the index data packet received in the queue, initiate a write request for updating the index data and a delete request for the original index data to the index service 103. Before initiating the write request for the index data, the reconstruction thread 604 can initiate a hold lock 11 for the corresponding primary key to the index service 103 and release the hold lock 11 after the index reconstruction of the primary key is successful.

[0078] Further, the control thread 602 can update the resource limit according to requirements by configuring and / or changing the resource limiter 605, so as to control the resources for the index rebuilding task.

[0079] Moreover, the control thread 602 can also control the start and stop of the index rebuilding task. Since the progress of the index rebuilding task of each scanning thread (i.e., the primary key of the last successful rebuilding) is saved in real time in the meta-information storage 601, after the task is stopped midway, the scanning progress can be resumed from the primary key of the last successful rebuilding.

[0080] It can be seen that according to the index rebuilding service-ification in the embodiments of the present disclosure, the index rebuilding task is performed through the interaction between the index rebuilding service 101, the index service 103, and the data service 102, so that the index rebuilding task can be paused, terminated, or resumed, realizing the overall controllability of the index rebuilding task. Moreover, through various function configurations in the index rebuilding service 101, the resource usage of the index rebuilding task can be restricted and adjusted in real time. This effectively avoids the uncontrollable impact on the performance of the database system during index rebuilding.

[0081] Based on the above index rebuilding method applied to the database system, the present disclosure also provides an index rebuilding device applied to the database system. The following will be combined with Figure 7 to describe this device in detail.

[0082] Figure 7 FIG. schematically shows a block diagram of an index rebuilding device 700 according to an embodiment of the present disclosure;

[0083] As Figure 7 shown, according to some embodiments of the present disclosure, the index rebuilding device 700 may include a scanning module 710, a rebuilding module 720, and a control module 730. According to some other embodiments of the present disclosure, the device 700 may further include a resource limiter 740 and / or a meta-information storage module 750. The index rebuilding device 700 can be used to provide the index rebuilding service 101 as Figure 1 shown.

[0084] The scanning module 710 includes at least one scanning thread, and can be used to read the stock data from the data service 102 through at least one scanning thread when there is a change in the index data structure of the stock data. In one embodiment, the scanning module 710 can perform the operation S310 described above.

[0085] The rebuilding module 720 includes at least one rebuilding thread, and is used to rebuild the index of the stock data in the index service 103 according to the changed index data structure through at least one rebuilding thread. In one embodiment, the rebuilding module 720 can perform the operation S320 described above.

[0086] The control module 730 includes a control thread, which is used to update and control the reconstruction progress of the index of the stock data in real time through the control thread during the process of reconstructing the index of the stock data. In one embodiment, the control module 730 may execute operation S330 described above.

[0087] The resource rate limiter 740 is used to set the currently allowed amount of scanned data according to the configuration of the control thread. Accordingly, the scanning module 710 is further used to, before each scanning thread scans the data service 102, apply to the resource rate limiter for the amount of data to be scanned this time through each scanning thread, and perform a scan of the data service 102 according to the amount of data to be scanned this time applied for through each scanning thread.

[0088] The meta - information storage module 750 is used to provide Figure 6 the meta - information storage function shown in, for specific reference to the above description, which will not be elaborated here.

[0089] According to an embodiment of the present disclosure, the index reconstruction device 700 can be used to implement the method described with reference to Figures 3 to 6 for specific reference to the above description, which will not be elaborated here.

[0090] According to an embodiment of the present disclosure, any multiple of the scanning module 710, the reconstruction module 720, the control module 730, the resource rate limiter 740, or the meta - information storage module 750 can be combined and implemented in one module, or any one of them can be split into multiple modules. Or, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the scanning module 710, the reconstruction module 720, the control module 730, the resource rate limiter 740, or the meta - information storage module 750 can be at least partially implemented as a hardware circuit, such as a field - programmable gate array (FPGA), a programmable logic array (PLA), a system - on - chip, a system - on - substrate, a system - on - package, an application - specific integrated circuit (ASIC), or can be implemented by any other reasonable way of integrating or packaging circuits, etc., in hardware or firmware, or implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Or, at least one of the scanning module 710, the reconstruction module 720, the control module 730, the resource rate limiter 740, or the meta - information storage module 750 can be at least partially implemented as a computer program module, which can execute the corresponding functions when the computer program module is run.

[0091] Figure 8 A block diagram of an electronic device suitable for implementing the index reconstruction method according to an embodiment of the present disclosure is schematically shown.

[0092] As Figure 8 shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage section 808 into a random access memory (RAM) 803. The processor 801 can include, for example, a general-purpose microprocessor (e.g., CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 can also include on-board memory for caching purposes. The processor 801 can include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0093] In the RAM 803, various programs and data required for the operation of the electronic device 800 are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. The processor 801 performs various operations of the method flow according to an embodiment of the present disclosure by executing the program in the ROM 802 and / or the RAM 803. It should be noted that the program can also be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 can also perform various operations of the method flow according to an embodiment of the present disclosure by executing the program stored in the one or more memories.

[0094] According to an embodiment of the present disclosure, the electronic device 800 can further include an input / output (I / O) interface 805, and the input / output (I / O) interface 805 is also connected to the bus 804. The electronic device 800 can further include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read from it can be installed into the storage section 808 as needed.

[0095] Embodiments of the present disclosure also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist alone without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the methods according to the embodiments of the present disclosure are implemented.

[0096] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the above-described ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803.

[0097] Embodiments of the present disclosure also include a computer program product, which includes a computer program that contains program code for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program code is used to cause the computer system to implement the method provided by the embodiments of the present disclosure.

[0098] When the computer program is executed by the processor 801, the above functions defined in the system / apparatus of the embodiments of the present disclosure are executed. According to an embodiment of the present disclosure, the above-described systems, apparatuses, modules, units, etc. may be implemented by computer program modules.

[0099] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and be downloaded and installed through the communication part 809, and / or be installed from the removable medium 811. The program code included in the computer program may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0100] In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiments of the present disclosure are executed. According to the embodiments of the present disclosure, the systems, devices, apparatuses, modules, units, etc. described above can be implemented by computer program modules.

[0101] According to the embodiments of the present disclosure, the program code for executing the computer programs provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The programming languages include, but are not limited to, such as Java, C++, Python, the "C" language, or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, by connecting through the Internet using an Internet service provider).

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combinations of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0103] Those skilled in the art can understand that the features recited in the various embodiments and / or claims of the present disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly recited in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features recited in the various embodiments and / or claims of the present disclosure can be combined and combined in various ways. All such combinations and / or combinations fall within the scope of the present disclosure.

[0104] The embodiments of the present disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments have been described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present disclosure.

Claims

1. An index rebuilding method applied to a database system, wherein, The database system includes a data service and an index service. The data service and the index service are respectively used to store and manage the original data and index data of the database system. An index reconstruction service is introduced to perform index reconstruction. Among them, the method includes: When there is a change in the index data structure of the stock data, the index reconstruction service reads the stock data from the data service; and Reconstructs the index of the stock data in the index service according to the changed index data structure; Among them, During the process of reconstructing the index of the stock data, through the configuration in the index reconstruction service, the reconstruction progress of the index of the stock data is updated and controlled in real time.

2. The method according to claim 1, wherein The reading of the stock data from the data service further includes: Reading the stock data through at least one scan. Among them, during each scan, the data is quantitatively read according to the data volume applied for this scan.

3. The method according to claim 2, wherein The reading of the stock data from the data service further includes: Applying for the data volume of this scan before each scan.

4. The method according to claim 1, wherein The reconstructing of the index of the stock data in the index service according to the changed index data structure further includes: Before reconstructing the index corresponding to any primary key in the index service according to the changed index data structure, a hold lock for the index corresponding to the primary key is initiated to the index service; among them, the hold lock is used to block the request of the data service to write data into the index corresponding to the primary key to the index service; and After the index corresponding to the primary key is reconstructed, the hold lock is released.

5. The method according to claim 1, wherein The real-time updating of the reconstruction progress of the index of the stock data includes: After the index corresponding to any primary key is reconstructed, the reconstruction progress is updated to the primary key name of the primary key.

6. The method according to claim 5, wherein, The controlling of the reconstruction progress of the index of the stock data includes: During the process of reconstructing the index of the stock data, when receiving an instruction to pause or resume the reconstruction of the index of the stock data, the reconstruction of the index of the stock data is paused or resumed.

7. The method according to claim 6, wherein The resuming of the reconstruction of the index of the stock data includes: Resuming the reconstruction starting from the index corresponding to the primary key after the primary key displayed in the reconstruction progress.

8. An index rebuilding device applied to a database system, wherein, The database system includes a data service and an index service. The data service and the index service are respectively used to store and manage the original data and index data of the database system. An index reconstruction service is introduced to perform index reconstruction. Among them, the device includes: A scanning module, including at least one scanning thread, which is used to, when there is a change in the index data structure of the stock data, read the stock data from the data service by the at least one scanning thread in the index reconstruction service; A reconstruction module, including at least one reconstruction thread, which is used to reconstruct the index of the stock data in the index service by the at least one reconstruction thread according to the changed index data structure; and A control module, including a control thread, which is used to, during the process of reconstructing the index of the stock data, update and control the reconstruction progress of the index of the stock data in real time by the control thread in the index reconstruction service.

9. The apparatus according to claim 8, wherein, The device further includes: A resource rate limiter for setting the currently allowed amount of scanned data according to the configuration of the control thread; Wherein, Before each scanning thread scans the data service, it applies to the resource rate limiter for the amount of data to be scanned this time, and each scanning thread scans the data service once according to the applied amount of data to be scanned this time.

10. An electronic device, comprising: One or more processors; A memory for storing one or more programs, Wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having executable instructions stored thereon, which when executed by a processor cause the processor to execute the method according to any one of claims 1 to 7.

12. A computer program product, comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images