Thread-safe index dynamic update method and system

Abstract

The present invention provides a thread-safe index dynamic update method and system, comprising: step S1: reading index file information in a directory to be indexed; step S2: initializing an index hash queue and an index status set according to the index file information, and establishing a dynamic monitoring instance; step S3: the dynamic monitoring instance monitors index changes according to the index status set within a preset interval, and when the index changes, step S4 is triggered, otherwise dynamic monitoring continues; step S4: checking the index hash queue, when the number of idle redundant indexes in the index hash queue is less than a threshold, a new index is created and added to the index hash queue, otherwise the idle redundant indexes in the index hash queue are directly changed.

Description

Technical Field

[0001] The present invention relates to the field of index technology, and in particular to a thread-safe index dynamic update method and system. Background Art

[0002] As the main underlying tool of search engines, Lucene is widely used in scenarios such as index creation, query and sorting. However, on the one hand, Lucene cannot dynamically perceive changes in index files, making it difficult to change index information in a timely manner. On the other hand, Lucene is thread-unsafe during index reconstruction or update, which can cause index conflicts in high-concurrency scenarios.

[0003] Patent document CN106294575A (application number: 201610604450.0) discloses a forest map spot vector data query method based on Lucene. The method includes the following steps: determining the grid size, converting the forest map spot vector data format, and gridding the converted data; establishing a full-text index for the gridded data, and returning the corresponding data according to the query conditions.

[0004] The prior art (page 77 of the journal "Industrial Control Computer" issue 33 in 2020 disclosed "An efficient index establishment method based on lucene") emphasizes the rapid establishment of indexes and the optimization of index structures. Although it can quickly establish indexes, it cannot guarantee real-time updates of indexes and the safety of update threads. Summary of the invention

[0005] In view of the defects in the prior art, the object of the present invention is to provide a thread-safe index dynamic update method and system.

[0006] A thread-safe index dynamic update method provided by the present invention includes:

[0007] Step S1: Read the index file information in the directory to be indexed;

[0008] Step S2: Initialize the index hash queue and index status set according to the index file information, and establish a dynamic monitoring instance;

[0009] Step S3: The dynamic monitoring instance monitors the index change according to the index status set within a preset interval. When the index changes, step S4 is triggered, otherwise the dynamic monitoring continues;

[0010] Step S4: Check the index hash queue. When the number of free redundant indexes in the index hash queue is less than the threshold, a new index is created and added to the index hash queue. Otherwise, the free redundant indexes in the index hash queue are directly changed.

[0011] Preferably, the step S2 adopts:

[0012] Step S2.1: According to the index files in the directory to be indexed, a file set Li = {x"} is obtained, where x" includes the file name and modification time;

[0013] Step S2.2: For each file in the file set Li, use the file name as the key ik, create a queue iq, add the index information ix to the queue, and put the queue iq into the index hash queue IM;

[0014] Step S2.3: Create a dynamic monitoring instance DM, and for each file in the file set Li, use the file name as the key sk and the file modification time as the value to put it into the index state set SL.

[0015] Preferably, the step S3 adopts:

[0016] Step S3.1: After the sleep time is up, the dynamic monitoring instance DM wakes up and scans the directory to be indexed to obtain a file set Lm = {x'}, where x' includes the file name and modification time;

[0017] Step S3.2: For each file in the index status set SL, if it does not exist in the file set Lm, it means that the index file has been deleted, and then delete the corresponding index queue in the index hash queue IM;

[0018] Step S3.3: For each index directory file of the file set Lm, if it does not exist in the index status set SL, use the file name as the key sk in the index status set and the file modification time as the value and place it in the index status set SL; if it exists in the index status set SL, compare the file modification time in the index status set SL. If the file modification time in the index status set SL is less than the modification time of the index directory file in Lm, update the modification time of the file in the index status set SL to the modification time of the index directory file in Lm, and add the file name to the file set Ln to be updated.

[0019] Preferably, the step S4 adopts:

[0020] Step S4.1: For each file to be updated in Ln, check whether the file to be updated in Ln exists in the index hash queue IM. If it exists, step S4.2 is triggered; if it does not exist, the file name is used as the key ik in the index hash queue, a queue iq is created, the index information ix is ​​queued, and the queue iq is placed in the index hash queue IM;

[0021] Step S4.2: Take out the index queue iq of the file from the index hash queue IM, reconstruct the index in according to the file, and when the number of elements in the index queue iq is less than the threshold, put in at the head of the queue; when the number of elements in the index queue iq is equal to the threshold, dequeue the tail element of the index queue iq, and put in at the head of the queue.

[0022] Preferably, the step S4 adopts: during the index query process, only the first index element of the iq team is queried, and other elements in iq are used as redundant indexes.

[0023] A thread-safe index dynamic update system provided by the present invention includes:

[0024] Module M1: read the index file information in the directory to be indexed;

[0025] Module M2: Initializes the index hash queue and index status set according to the index file information, and establishes a dynamic monitoring instance;

[0026] Module M3: Dynamic monitoring instance monitors index changes according to the index status set within a preset interval. When the index changes, module M4 is triggered, otherwise dynamic monitoring continues;

[0027] Module M4: Check the index hash queue. When the number of free redundant indexes in the index hash queue is less than the threshold, a new index is created and added to the index hash queue. Otherwise, the free redundant indexes in the index hash queue are directly changed.

[0028] Preferably, the module M2 adopts:

[0029] Module M2.1: According to the index files in the directory to be indexed, obtain the file set Li = {x"} of the initialization index directory, where x" includes the file name and modification time;

[0030] Module M2.2: For each initialization index file of Li, use the file name as the key ik in the index hash queue, create a queue iq, enter the index information ix into the queue, and put the queue iq into the index hash queue IM;

[0031] Module M2.3: Create a dynamic monitoring instance DM, and for each initialized index file of Li, put it into the index state set SL with the file name as the key sk in the index state set and the file modification time as the value.

[0032] Preferably, the module M3 adopts:

[0033] Module M3.1: After the sleep time is up, the dynamic monitoring instance DM wakes up and scans the directory to be indexed to obtain the file set Lm = {x'} in the current index directory, where x' contains the file name and modification time;

[0034] Module M3.2: For each file in the index status set SL, if it does not exist in the file set Lm in the current index directory, it means that the index file has been deleted, then delete the corresponding index queue in the index hash queue IM;

[0035] Module M3.3: For each index directory file of Lm, if it does not exist in the index status set SL, use the file name as the key sk in the index status set and the file modification time as the value and place it in the index status set SL; if it exists in the index status set SL, compare the file modification time in the index status set SL. If the file modification time in the index status set SL is less than the modification time of the index directory file in Lm, update the modification time of the file in the index status set SL to the modification time of the index directory file in the file set Lm, and add the file name to the file set Ln of the index to be updated;

[0036] Preferably, the module M4 adopts:

[0037] Module M4.1: For each file to be updated in Ln, check whether the file to be updated in Ln exists in the index hash queue IM. If it exists, module M4.2 is triggered; if it does not exist, the file name is used as the key ik in the index hash queue, a queue iq is created, the index information ix is ​​queued, and the queue iq is placed in the index hash queue IM;

[0038] Module M4.2: Take out the index queue iq of the file from the index hash queue IM, reconstruct the index in according to the file, and when the number of elements in the index queue iq is less than the threshold, put in at the head of the queue; when the number of elements in the index queue iq is equal to the threshold, dequeue the tail element of the index queue iq, and put in at the head of the queue.

[0039] Preferably, the module M4 adopts: during the index query process, only the head index element of the iq team is queried, and other elements in iq are used as redundant indexes.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The present invention uses a dynamic monitoring method to check the modification time of the index file and compare it with the cached index status set, thereby solving the problem of dynamic perception when the index changes in the index directory;

[0042] 2. The present invention adopts the structure of index hash queue. When the index changes, the idle redundant index at the end of the queue is updated, and the updated redundant index is moved to the head of the queue as the active index. On the one hand, it solves the problem of dynamic update of the index, and on the other hand, it solves the problem of thread insecurity caused by changing the active index during the process of updating the index. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0044] Figure 1 Flowchart of thread-safe index dynamic update method.

[0045] Figure 2 Flowchart of the method for checking the index hash queue.

[0046] Figure 3 Schematic diagram of a thread-safe index dynamic update system in an embodiment. DETAILED DESCRIPTION

[0047] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0048] Example 1

[0049] The present invention uses dynamic monitoring technology to monitor changes in index files in the directory to be indexed, compares them with the current index version and time, and determines whether the index needs to be updated.

[0050] When the index needs to be updated, the method can automatically rebuild the index and ensure that the index user can still use the index normally during the index rebuilding process. The thread-safe index update method proposed by the present invention can ensure the availability and thread safety of the index in the scenario of high concurrent query when the index is dynamically changed.

[0051] According to a thread-safe index dynamic update method provided by the present invention, Figures 1 to 2 As shown, including:

[0052] Step S1: Read the index file information in the directory to be indexed;

[0053] Step S2: Initialize the index hash queue and index status set according to the index file information, and establish a dynamic monitoring instance;

[0054] Step S3: The dynamic monitoring instance monitors the index change according to the index status set within a preset interval. When the index changes, step S4 is triggered, otherwise the dynamic monitoring continues;

[0055] Step S4: Check the index hash queue. When the number of free redundant indexes in the index hash queue is less than the threshold, a new index is created and added to the index hash queue. Otherwise, the free redundant indexes in the index hash queue are directly changed.

[0056] Specifically, the step S2 adopts:

[0057] Input: path of directory to be indexed;

[0058] Output: index queue iq, dynamic monitoring instance DM;

[0059] Step S2.1: Scan the index files in the directory to be indexed to obtain a file set Li = {x"}, where x" includes the file name and modification time;

[0060] Step S2.2: For each file in the file set Li, use the file name as the key ik, create a queue iq, add the index information ix to the queue, and put the queue iq into the index hash queue IM;

[0061] Step S2.3: Create a dynamic monitoring instance DM, and for each file in the file set Li, use the file name as the key sk and the file modification time as the value to put it into the index state set SL.

[0062] Specifically, the step S3 adopts:

[0063] Input: path of directory to be indexed;

[0064] Output: the file set that needs to be updated Ln = {x};

[0065] Step S3.1: After the sleep time is up, the dynamic monitoring instance DM wakes up and scans the directory to be indexed to obtain a file set Lm = {x'}, where x' includes the file name and modification time;

[0066] Step S3.2: For each file in the index status set SL, if it does not exist in the file set Lm, it means that the index file has been deleted, and then delete the corresponding index queue in the index hash queue IM;

[0067] Step S3.3: For each file in the file set Lm, if the file does not exist in the index status set SL, place it in the index status set SL with the file name as the key sk and the file modification time as the value; if it exists in the index status set SL, compare the file modification time in the index status set SL. If the file modification time in the index status set SL is less than the file modification time, update the file modification time in the index status set SL to the file modification time in the file set Lm, and add the file name to the file set Ln that needs to be updated.

[0068] Specifically, the step S4 adopts:

[0069] Input: the file set to be updated Ln = {x};

[0070] Output: updated index hash queue IM;

[0071] Step S4.1: For each file in the file set Ln, check whether the file to be updated in the file set Ln exists in the index hash queue IM. If it does, step S4.2 is triggered; if it does not exist, the file name is used as the key ik, a queue iq is created, the index information ix is ​​queued, and the queue iq is placed in the index hash queue IM;

[0072] Step S4.2: Take the index queue iq of the file from the index hash queue IM, reconstruct the index in according to the index file, and when the number of elements in the index queue iq is less than the threshold (self-set according to the concurrency and memory size, generally 3), put the in queue at the head of the queue; when the number of elements in the index queue iq is equal to the threshold, remove the tail element of the index queue iq, and put the in queue at the head of the queue;

[0073] Specifically, the step S4 adopts: during the index query process, only the first index element of the iq team is queried, and other elements in iq are used as redundant indexes.

[0074] A thread-safe index dynamic update system provided by the present invention includes:

[0075] Module M1: read the index file information in the directory to be indexed;

[0076] Module M2: Initializes the index hash queue and index status set according to the index file information, and establishes a dynamic monitoring instance;

[0077] Module M3: Dynamic monitoring instance monitors index changes according to the index status set within a preset interval. When the index changes, module M4 is triggered, otherwise dynamic monitoring continues;

[0078] Module M4: Check the index hash queue. When the number of free redundant indexes in the index hash queue is less than the threshold, a new index is created and added to the index hash queue. Otherwise, the free redundant indexes in the index hash queue are directly changed.

[0079] Specifically, the module M2 adopts:

[0080] Input: path of directory to be indexed;

[0081] Output: index queue iq, dynamic monitoring instance DM;

[0082] Module M2.1: Scan the QR code to index the files in the directory to be indexed, and obtain the file set Li = {x”}, where x” includes the file name and modification time;

[0083] Module M2.2: For each file in the file set Li, use the file name as the key ik, create a queue iq, add the index information ix to the queue, and put the queue iq into the index hash queue IM;

[0084] Module M2.3: Create a dynamic monitoring instance DM, and for each file in the file set Li, use the file name as the key sk and the file modification time as the value and put it into the index status set SL.

[0085] Specifically, the module M3 adopts:

[0086] Input: path of directory to be indexed;

[0087] Output: the file set that needs to be updated Ln = {x};

[0088] Module M3.1: After the sleep time is up, the dynamic monitoring instance DM wakes up and scans the directory to be indexed to obtain the file set Lm = {x'}, where x' contains the file name and modification time;

[0089] Module M3.2: For each file in the index status set SL, if it does not exist in the file set Lm, it means that the index file has been deleted, then delete the corresponding index queue in the index hash queue IM;

[0090] Module M3.3: For each file in the file set Lm, if the file does not exist in the index status set SL, it is placed in the index status set SL with the file name as the key sk and the file modification time as the value; if it exists in the index status set SL, the file modification time in the index status set SL is compared. If the file modification time in the index status set SL is less than the file modification time, the file modification time in the index status set SL is updated to the file modification time in the file set Lm, and the file name is added to the file set Ln that needs to be updated.

[0091] Specifically, the module M4 adopts:

[0092] Input: the file set to be updated Ln = {x};

[0093] Output: updated index hash queue IM;

[0094] Module M4.1: For each file x in the file set Ln, check whether the file to be updated in the index in Ln exists in the index hash queue IM. If it does, module M4.2 is triggered; if it does not exist, the file name is used as the key ik, a queue iq is created, the index information ix is ​​queued, and the queue iq is placed in the index hash queue IM;

[0095] Module M4.2: Take the index queue iq of the file from the index hash queue IM, reconstruct the index in according to the index file, and when the number of elements in the index queue iq is less than the threshold (self-set according to the concurrency and memory size, generally 3), put in at the head of the queue; when the number of elements in the index queue iq is equal to the threshold, remove the tail element of the index queue iq and put in at the head of the queue;

[0096] Specifically, the module M4 adopts: during the index query process, only the first index element of the iq team is queried, and other elements in iq are used as redundant indexes.

[0097] Example 2

[0098] Embodiment 2 is a preferred embodiment of Embodiment 1

[0099] Figure 3 As an example of a dynamic indexing system, the thread-safe index dynamic update system provides two query methods: user access query and external system call query. After receiving the user or system behavior, the system distributes the request to the index management use case through the index query use case. The index management use case maintains a dynamically changing index queue to accept user requests and dynamically update index information. The index update use case monitors the index update status through the index file management use case. If the index is updated, a new index will be dynamically created and the index management use case will be notified to update. After receiving the notification, the index management use case uses a thread-safe method to update the index file. The system timing steps are as follows:

[0100] S1: Index file management use case reads and parses index files;

[0101] S2: The index initialization use case initializes the index information and maintains the index through the index management use case. The system is initialized and accepts user requests.

[0102] S3: The end user or external system communicates with the system through the two sub-use cases of index query to query index information;

[0103] S4: The index query use case calls the index management use case and returns the query results to the user;

[0104] S5: The index update use case monitors the index update status in real time, dynamically updates the index information and notifies the index management use case;

[0105] S6: The index management use case updates the global index after receiving the notification.

[0106] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0107] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A thread-safe index dynamic update method, characterized in that: include: Step S1: Read the index file information in the directory to be indexed; Step S2: Initialize the index hash queue and index status set according to the index file information, and establish a dynamic monitoring instance; Step S3: The dynamic monitoring instance monitors the index change according to the index status set within a preset interval. When the index changes, step S4 is triggered, otherwise the dynamic monitoring continues; Step S4: Check the index hash queue. When the number of free redundant indexes in the index hash queue is less than the threshold, create a new index and add it to the index hash queue. Otherwise, directly change the free redundant indexes in the index hash queue. The step S3 adopts: Input: path of directory to be indexed; Output: the file set Ln to be updated index = {x}; Step S3.1: After the sleep time is up, the dynamic monitoring instance DM wakes up and scans the directory to be indexed to obtain the file set Lm={x'} in the current index directory, where x' includes the file name and modification time; Step S3.2: For each file in the index status set SL, if it does not exist in the file set Lm in the current index directory, it means that the index file has been deleted, and the corresponding index queue in the index hash queue IM is deleted; Step S3.3: For each index directory file of Lm, if it does not exist in the index status set SL, use the file name as the key sk in the index status set and the file modification time as the value and place it in the index status set SL; if it exists in the index status set SL, compare the file modification time in the index status set SL. If the file modification time in the index status set SL is less than the modification time of the index directory file in Lm, update the modification time of the file in the index status set SL to the modification time of the index directory file in Lm, and add the file name to the file set Ln of the index to be updated; The step S4 adopts: Input: the file set Ln to be updated index = {x}; Output: updated index hash queue IM'; Step S4.1: For each file to be updated in Ln, check whether the file to be updated in Ln exists in the index hash queue IM. If it exists, step S4.2 is triggered; if it does not exist, the file name is used as the key ik in the index hash queue, a queue iq is created, the index information ix is ​​queued, and the queue iq is placed in the index hash queue IM; Step S4.2: Take out the index queue iq of the file from the index hash queue IM, reconstruct the index in according to the file, and when the number of elements in the index queue iq is less than the threshold, put in at the head of the queue; when the number of elements in the index queue iq is equal to the threshold, dequeue the tail element of the index queue iq, and put in at the head of the queue.

2. The thread-safe index dynamic update method according to claim 1, characterized in that: The step S2 adopts: Step S2.1: According to the index files in the directory to be indexed, obtain a file set Li={x”} of the initialization index directory, where x” includes the file name and modification time; Step S2.2: For each initialized index file of Li, use the file name as the key ik in the index hash queue, create a queue iq, add the index information ix to the queue, and put the queue iq into the index hash queue IM; Step S2.3: Create a dynamic monitoring instance DM, and for each initialized index file of Li, put it into the index state set SL with the file name as the key sk in the index state set and the file modification time as the value.

3. The thread-safe index dynamic update method according to claim 1, characterized in that: The step S4 adopts: during the index query process, only the first index element of the iq team is queried, and other elements in iq are used as redundant indexes.

4. A thread-safe index dynamic update system, characterized in that: include: Module M1: read the index file information in the directory to be indexed; Module M2: Initializes the index hash queue and index status set according to the index file information, and establishes a dynamic monitoring instance; Module M3: Dynamic monitoring instance monitors index changes according to the index status set within a preset interval. When the index changes, module M4 is triggered, otherwise dynamic monitoring continues; Module M4: Check the index hash queue. When the number of free redundant indexes in the index hash queue is less than the threshold, a new index is created and added to the index hash queue. Otherwise, the free redundant indexes in the index hash queue are directly changed. The module M3 adopts: Input: path of directory to be indexed; Output: the file set Ln to be updated index = {x}; Module M3.1: After the sleep time is up, the dynamic monitoring instance DM wakes up and scans the directory to be indexed to obtain the file set Lm = {x'} in the current index directory, where x' contains the file name and modification time; Module M3.2: For each file in the index status set SL, if it does not exist in the file set Lm in the current index directory, it means that the index file has been deleted, then delete the corresponding index queue in the index hash queue IM; Module M3.3: For each index directory file of Lm, if it does not exist in the index status set SL, use the file name as the key sk in the index status set and the file modification time as the value and place it in the index status set SL; if it exists in the index status set SL, compare the file modification time in the index status set SL. If the file modification time in the index status set SL is less than the modification time of the index directory file in Lm, update the modification time of the file in the index status set SL to the modification time of the index directory file in the file set Lm, and add the file name to the file set Ln of the index to be updated; The module M4 adopts: Input: the file set Ln to be updated index = {x}; Output: updated index hash queue IM'; Module M4.1: For each file to be updated in Ln, check whether the file to be updated in Ln exists in the index hash queue IM. If it exists, module M4.2 is triggered; if it does not exist, the file name is used as the key ik in the index hash queue, a queue iq is created, the index information ix is ​​queued, and the queue iq is placed in the index hash queue IM; Module M4.2: Take out the index queue iq of the file from the index hash queue IM, reconstruct the index in according to the file, and when the number of elements in the index queue iq is less than the threshold, put in at the head of the queue; when the number of elements in the index queue iq is equal to the threshold, dequeue the tail element of the index queue iq, and put in at the head of the queue.

5. The thread-safe index dynamic update system according to claim 4, characterized in that: The module M2 adopts: Module M2.1: According to the index files in the directory to be indexed, obtain the file set Li = {x"} of the initialization index directory, where x" includes the file name and modification time; Module M2.2: For each initialization index file of Li, use the file name as the key ik in the index hash queue, create a queue iq, enter the index information ix into the queue, and put the queue iq into the index hash queue IM; Module M2.3: Create a dynamic monitoring instance DM, and for each initialized index file of Li, put it into the index state set SL with the file name as the key sk in the index state set and the file modification time as the value.

6. The thread-safe index dynamic update system according to claim 4, characterized in that: The module M4 adopts: during the index query process, only the first index element of the iq team is queried, and other elements in iq are used as redundant indexes.

Images