Method and system for improving Lucene performance in Elasticsearch cluster based on read-write lock mechanism

Abstract

The invention provides a method and a system for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism, and relates to the technical field of databases. According to the method, a novel more flexible and efficient mode is provided besides the traditional scheme for processing the concurrent access efficiency of the shared resources in the Elasticsearch cluster, and the concurrent access of the shared resources is processed through a read-write lock mechanism. According to the invention, a plurality of threads are allowed to read data from a cache at the same time, meanwhile, the exclusiveness of write operation is ensured, the performance of the whole Elasticsearch cluster is improved, and the response speed of read-write operation is improved.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a method and system for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism. Background Art

[0002] In enterprises, Elasticsearch is often used to build search engines that support complex queries and filtering, and can quickly perform full-text searches in large document collections. However, as enterprises grow larger and larger, the amount of data increases, and the corresponding scene usage and complex read and write operations become more frequent. Therefore, Lucene will also face performance challenges under high load conditions. One of the most significant problems is lock contention in the cache system. When the query volume increases, it will drag down the performance of the entire Elasticsearch cluster, resulting in a decline in response speed and search efficiency. Summary of the invention

[0003] Purpose of the invention: To propose a method for improving the performance of Lucene in an Elasticsearch cluster based on a read-write lock mechanism, and to propose a system for implementing the method, so as to solve the above problems existing in the prior art.

[0004] In a first aspect of the present invention, a method for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism is proposed, and the steps are as follows: S1: The client sends a read / write request to the Elasticsearch cluster. This request can be sent to any node in the cluster. If the node receives the read / write request, it enters S2. Otherwise, an exception is thrown. S2: The node that receives the request will act as a coordinator. The coordinator is responsible for processing the client's request and routing the request to the correct data node and entering S3. Otherwise, an exception is thrown; S3: The coordination node determines which primary shard the document should be written to based on the document ID and index settings (such as the number of shards). This is achieved through a hash function and modulo operation to ensure that documents with the same _id are always routed to the same primary shard. Once the primary shard is successfully determined, it proceeds to S4. Otherwise, an exception is thrown. S4: After determining the target primary shard, the coordination node forwards the request to the data node where the primary shard is located. After receiving the request, the primary shard on the data node will first write the document into the Lucene index structure in the memory and perform index inversion. If successful, it will enter S5 and S6. Otherwise, an exception will be thrown. S5: After the primary shard on the data node receives the request, the Lucene index selects different lock strategies according to the content of the read and write operations. The content of the read and write operations needs to match the corresponding scenarios, such as allowing multiple threads to lock the same data, allowing one thread to lock the data, and controlling and sharing concurrent access to resources. If the read and write lock strategy is successfully executed, it enters S6, otherwise, an exception is thrown; S6: The inverted index maps the words in the document to the list of documents containing these words, so as to achieve fast search and retrieval. After the inverted index is successfully executed, the data is stored in the data node. If successful, it enters S7, otherwise it enters S1 and throws an exception; S7: After the inverted index is completed, the coordination node aggregates the results from each data node, sorts them, and performs other processing, and finally returns the results to the client; S8: End the process.

[0005] In a further embodiment, in step S5, different lock strategies are selected according to the contents of the read and write operations. There are three types in total: Concurrent read access: Multiple queries can read data from the cache at the same time, reducing wait time and increasing overall throughput.

[0006] Controlling write access: Write operations still require an exclusive lock to ensure data integrity during updates.

[0007] Reduce lock contention: By allowing concurrent reads, the frequency and severity of lock contention are significantly reduced.

[0008] In a further embodiment, in step S5, the scenes are divided into three types; Vector retrieval scenario: As a vector database, Elasticsearch requires efficient processing and complex operations when performing tasks such as similarity search.

[0009] BM25 scoring scenario: As a popular ranking function in information retrieval, BM25 scoring is directly related to low latency and high throughput, and is suitable for requirements on the relevance and speed of search results.

[0010] Range query scenario: This involves scanning a large portion of the range query index and requires high performance.

[0011] In a further embodiment, in step S1, the request format may be xml, json and support jsonp cross-domain access.

[0012] In a further embodiment, the management of read-write locks is divided into two ways: Timeout mechanism: To prevent threads from waiting for locks for a long time, you can set a timeout for acquiring locks. If the lock cannot be acquired within the timeout, an error is returned or other strategies are adopted.

[0013] Lock upgrade and downgrade: Depending on the actual situation, the read lock can be upgraded to a write lock, or the write lock can be downgraded to a read lock. For example, when a read thread needs to perform a write operation, the read lock can be upgraded to a write lock; when a write thread completes the write operation, the write lock can be downgraded to a read lock to allow other read and write operations to continue.

[0014] In a second aspect of the present invention, a system for improving the performance of Lucene in an Elasticsearch cluster based on a read-write lock mechanism is proposed. The system includes a client, a first response module, a second response module, a third response module, a fourth response module, a fifth response module, and a sixth response module.

[0015] The client sends a read / write request to any node in the Elasticsearch cluster. If the node receives the read / write request, it will feedback to the first response module; otherwise, an exception will be thrown. The first response module controls the node that receives the request as a coordination node, routes the read / write request to the correct data node, and feeds back to the second response module; otherwise, an exception is thrown; The second response module controls the coordination node to determine the target primary shard to be written according to the document ID and index settings. After the target primary shard is successfully determined, feedback is sent to the third response module; otherwise, an exception is thrown; The third response module controls the coordination node to forward the read / write request to the data node where the primary shard is located. After receiving the read / write request, the primary shard on the data node first writes the document into the Lucene index structure in the memory and feeds back to the fourth response module; otherwise, an exception is thrown; After the primary shard on the data node receives the read / write request, the fourth response module controls the Lucene index to select and execute different read / write lock strategies according to the content of the read / write operation. The content of the read / write operation matches the corresponding scenario. If the read / write lock strategy is successfully executed, it will be fed back to the fifth response module; otherwise, an exception will be thrown; The fifth response module maps the words in the document to a list of documents containing the words; after the inverted index is successfully executed, the data is stored in the data node, and feedback is sent to the sixth response module if successful; otherwise, feedback is sent to the client and an exception is thrown; After the inverted index is completed, the sixth response module coordinates the node to aggregate the results from each data node, sort and paginate them, and finally returns the results to the client.

[0016] In a third aspect of the present invention, an electronic device is proposed, comprising a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism as described in the first aspect is implemented.

[0017] In a fourth aspect of the present invention, a computer-readable storage medium is proposed, which stores at least one executable instruction. When the executable instruction is executed on an electronic device, the electronic device executes the method for improving the performance of Lucene in an Elasticsearch cluster based on a read-write lock mechanism as described in the first aspect.

[0018] Beneficial effects: In addition to the traditional solution for improving the efficiency of concurrent access to shared resources in the Elasticsearch cluster, the present invention provides a new, more flexible and efficient way to handle concurrent access to shared resources through a read-write lock mechanism. It allows multiple threads to read data from the cache at the same time while ensuring the exclusivity of write operations, improving the performance of the entire Elasticsearch cluster and increasing the response speed of read and write operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flowchart of a method for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism. DETAILED DESCRIPTION

[0020] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0021] This embodiment discloses a method for improving the performance of Lucene in an Elasticsearch cluster based on a read-write lock mechanism: a client sends a write request to an Elasticsearch cluster, and this request can be sent to any node in the cluster. The node that receives the request will act as a coordinator node, which is responsible for processing the client's request and routing the request to the correct data node. The coordinator node determines which primary shard the document should be written to based on the document ID and index settings (such as the number of shards). This is achieved through a hash function and modulo operation to ensure that documents with the same _id are always routed to the same primary shard. After determining the target primary shard, the coordinator node forwards the request to the data node where the primary shard is located. After receiving the request, the primary shard on the data node will first write the document to the Lucene index structure in memory. This process includes converting the document into an inverted index for subsequent search and analysis. Elasticsearch uses Lucene as its underlying search library. Lucene builds an inverted index to speed up the search process. The inverted index maps words in a document to a list of documents containing these words, thereby achieving fast search and retrieval. The data nodes execute the query locally and return the results to the coordination node. The coordination node aggregates the results from each data node, sorts, and paginates, and finally returns the results to the client. When users index low cardinality fields, such as fixed values, they usually use numbers to represent these values. Elasticsearch infers these fields as "long" data types and uses BKD trees as indexes for these long fields. As the amount of data grows, building result sets for these low cardinality fields may cause high CPU usage and increase system load. A more flexible and efficient way to handle concurrent access to shared resources based on the read-write lock mechanism is needed, allowing multiple threads to read data from the cache at the same time while ensuring the exclusivity of write operations, ensuring the reliability and robustness of Elasticsearch, improving the responsiveness of the Elasticsearch cluster, and improving search efficiency. 1) This problem becomes more obvious when the CPU is overloaded in batch data operations. During the re-indexing process, converting the "long" field to the "keyword" field can significantly reduce cluster load and query latency, but this process may take a long time. 2) Considering the response speed and search efficiency, it is best for the client to use the "keyword" field when performing term / terms queries and the "long" field when performing range queries.However, in actual situations, users often fail to realize the performance impact of using "long" fields on low-cardinality fields and rely on the data type automatically selected by dynamic mapping. Therefore, optimizing the BKD tree based on the read-write lock mechanism can solve the low / medium cardinality field problem, ultimately effectively reducing CPU load and query latency.

[0022] Specifically, see Figure 1 As shown in the figure, the method steps for improving the performance of Lucene in the Elasticsearch cluster based on the read-write lock mechanism are as follows: S1: The client sends a read / write request to the Elasticsearch cluster. The request format can be xml or json and supports jsonp cross-domain access. This request can be sent to any node in the cluster. If the node receives the read / write request, it enters S2. Otherwise, an exception is thrown. S2: The node that receives the request will act as a coordinator. The coordinator is responsible for processing the client's request and routing the request to the correct data node and entering S3. Otherwise, an exception is thrown.

[0023] S3: The coordination node determines which primary shard the document should be written to based on the document's _id and index settings (such as the number of shards). This is achieved through a hash function and modulo operation to ensure that documents with the same _id are always routed to the same primary shard. Once the primary shard is successfully determined, it proceeds to S4. Otherwise, an exception is thrown. S4: After determining the target primary shard, the coordination node forwards the request to the data node where the primary shard is located. After receiving the request, the primary shard on the data node will first write the document into the Lucene index structure in the memory and perform index inversion. If successful, it will enter S5 and S6. Otherwise, an exception will be thrown. S5: After the primary shard on the data node receives the request, the Lucene index selects different lock strategies according to the content of the read and write operations. The content of the read and write operations needs to match the corresponding scenarios, such as allowing multiple threads to lock the same data, allowing one thread to lock the data, and controlling and sharing concurrent access to resources. If the read and write lock strategy is successfully executed, it enters S6, otherwise, an exception is thrown; S6: The inverted index maps the words in the document to the list of documents containing these words, so as to achieve fast search and retrieval. After the inverted index is successfully executed, the data is stored in the data node. If successful, it enters S7, otherwise it enters S1 and throws an exception; S7: After the inverted index is completed, the coordination node aggregates the results from each data node, sorts them, and performs other processing, and finally returns the results to the client; S8: End the process.

[0024] As a preferred embodiment, there are three types of lock strategies: Concurrent read access: Multiple queries can read data from the cache at the same time, reducing wait time and increasing overall throughput.

[0025] Controlling write access: Write operations still require an exclusive lock to ensure data integrity during updates.

[0026] Reduce lock contention: By allowing concurrent reads, the frequency and severity of lock contention are significantly reduced.

[0027] As a preferred embodiment, the scenarios are divided into 3 types: Vector retrieval scenario: As a vector database, Elasticsearch requires efficient processing and complex operations when performing tasks such as similarity search.

[0028] BM25 scoring scenario: As a popular ranking function in information retrieval, BM25 scoring is directly related to low latency and high throughput, and is suitable for requirements on the relevance and speed of search results.

[0029] Range query scenario: This involves scanning a large portion of the range query index and requires high performance.

[0030] As a preferred embodiment, the management of read-write locks is divided into two ways: Timeout mechanism: To prevent threads from waiting for locks for a long time, you can set a timeout for acquiring locks. If the lock cannot be acquired within the timeout, an error is returned or other strategies are adopted.

[0031] Lock upgrade and downgrade: Depending on the actual situation, the read lock can be upgraded to a write lock, or the write lock can be downgraded to a read lock. For example, when a read thread needs to perform a write operation, the read lock can be upgraded to a write lock; when a write thread completes the write operation, the write lock can be downgraded to a read lock to allow other read and write operations to continue.

[0032] As a preferred embodiment, the Elasticsearch cluster uses a read-write lock mechanism in four steps: Step 1: System initialization When the Elasticsearch cluster is started, the read-write lock mechanism is initialized, a read-write lock is allocated to each Lucene index, and the performance monitoring module is started to monitor the system performance in real time.

[0033] Step 2: Read operation processing When a thread needs to read Lucene index data, it acquires a read lock. If other threads are performing read operations, the new read thread can immediately acquire the read lock without waiting. Data is read from the index and stored in the cache so that subsequent read operations can directly obtain data from the cache, reducing repeated reading of the index.

[0034] Step 3: Write operation processing When a thread needs to write Lucene index data, it acquires the write lock. If other threads are performing read or write operations, the write thread needs to wait until it acquires the write lock. Collect multiple write requests and process them in batches. When the batch reaches a certain size or after a certain period of time, write to the index at one time to reduce lock contention and disk I / O operations.

[0035] Step 4: Lock management and optimization To avoid deadlock, set a timeout when acquiring a lock. If a thread fails to acquire a lock within the timeout, the operation is abandoned and an error message is returned. Regularly check and clean up unused locks to release system resources. For example, you can set a timer to check the usage of a lock at regular intervals and release it if a lock has not been used for a certain period of time.

[0036] In an enterprise's internal log analysis system, the Elasticsearch cluster is used to store and analyze a large amount of log data. By introducing the read-write lock mechanism and optimization module of the present invention, the system's performance is significantly improved when processing a large number of log write and query operations. At the same time, the stability and reliability of the system are also improved, reducing system failures and performance degradation caused by lock competition.

[0037] As a preferred embodiment, a system for improving the performance of Lucene in an Elasticsearch cluster based on a read-write lock mechanism is also disclosed. The system includes a client, a first response module, a second response module, a third response module, a fourth response module, a fifth response module, and a sixth response module.

[0038] The client sends a read / write request to any node in the Elasticsearch cluster. If the node receives the read / write request, it will be fed back to the first response module; otherwise, an exception will be thrown. The first response module controls the node that receives the request as a coordination node, routes the read / write request to the correct data node, and feeds back to the second response module; otherwise, an exception will be thrown. The second response module controls the coordination node to determine the target primary shard to be written according to the document ID and index settings. After the target primary shard is successfully determined, it will be fed back to the third response module; otherwise, an exception will be thrown. The third response module controls the coordination node to forward the read / write request to the data node where the primary shard is located. After the primary shard on the data node receives the read / write request, it first writes the document into the Lucene index structure in the memory and feeds back to the fourth response module; otherwise, an exception will be thrown. After the primary shard on the data node receives the read / write request, the fourth response module controls the Lucene index to select and execute different read / write lock strategies according to the content of the read / write operation. The content of the read / write operation matches the corresponding scenario. After the read / write lock strategy is successfully executed, it will be fed back to the fifth response module; otherwise, an exception will be thrown. The fifth response module maps the words in the document to the list of documents containing the words; after the inverted index is successfully executed, the data is stored in the data node, and feedback is sent to the sixth response module after success; otherwise, feedback is sent to the client and an exception is thrown. After the inverted index is completed, the sixth response module coordinates the node to aggregate the results from each data node, sort and paginate, and finally return the results to the client.

[0039] When the above system is running, it can execute a method for improving the performance of Lucene in the Elasticsearch cluster based on a read-write lock mechanism. The specific process is described in the above embodiment and will not be repeated here.

[0040] In practical applications, the operation logic of the method for improving the performance of Lucene in an Elasticsearch cluster based on a read-write lock mechanism disclosed in the above embodiment can also be written into a set of executable instructions, which can be written into a storage medium and run on an electronic device.

[0041] The read operation lock script is as follows: # Read operation lock script POST _scripts / rw-lock { "script": { "lang": "painless", "source": "if (ctx._source.lock_type == 'exclusive') {Debug.explain('one thread is writing data, the lock is exclusive now');}ctx._source.lock_count++" } } # Release the lock script after the read operation is completed POST _scripts / rw-unlock { "script": { "lang": "painless", "source": "if ( --ctx._source.lock_count == 0) { ctx.op = 'delete'}" } } More specific examples of the computer-readable storage medium mentioned in this embodiment may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device.

[0042] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism, characterized in that: The steps include: S1: The client sends a read / write request to any node in the Elasticsearch cluster. If the node receives the read / write request, it enters S2. Otherwise, throw an exception; S2: The node that receives the request acts as a coordinating node, routes the read / write request to the correct data node, and enters S3; Otherwise, throw an exception; S3: The coordinating node determines the target primary shard to be written based on the document ID and index settings. After the target primary shard is successfully determined, it enters S4; Otherwise, throw an exception; S4: The coordinating node forwards the read / write request to the data node where the primary shard is located. After receiving the read / write request, the primary shard on the data node first writes the document into the Lucene index structure in memory and enters S5. Otherwise, throw an exception; S5: After the primary shard on the data node receives the read / write request, the Lucene index selects and executes different read / write lock strategies according to the content of the read / write operation. The content of the read / write operation matches the corresponding scenario. If the read / write lock strategy is successfully executed, it enters S6; otherwise, an exception is thrown. S6: Map the words in the document to the list of documents containing the words, so as to achieve fast search and retrieval. After the inverted index is successfully executed, the data is stored in the data node. If successful, enter S7; otherwise, return to S1 and throw an exception; S7: After the inverted index is completed, the coordination node aggregates the results from each data node, sorts and paginates them, and finally returns the results to the client; S8: End the process.

2. The method for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism according to claim 1, characterized in that: In step S1, the format of the read and write request is xml, json, and supports jsonp cross-domain access.

3. The method for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism according to claim 1, characterized in that: In step S5, the Lucene index selects and executes different read-write lock strategies according to the content of the read and write operations, including the following read-write lock strategies: Concurrent read access: multiple queries read data from the cache at the same time; Controlling write access: write operations still require an exclusive lock; Reduce lock contention: allow concurrent reads.

4. The method for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism according to claim 1, characterized in that: In step S5, the content of the read and write operations is matched with the corresponding scenarios, which are classified into the following types: Vector retrieval scenario: Elasticsearch, as a vector database, requires efficient processing and complex operations when performing similarity search tasks; BM25 score scenario: As a popular ranking function in information retrieval, BM25 score is directly related to low latency and high throughput, and is suitable for requirements on the relevance and speed of search results; Range query scenario: This involves range queries that scan large parts of the index and require higher performance.

5. The method for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism according to claim 1, characterized in that: The read-write lock includes the following two management modes: Timeout mechanism: Set the lock acquisition timeout. If the lock cannot be acquired within the timeout, an error is returned or other strategies are adopted. Lock upgrade and downgrade: Depending on the actual situation, the read lock is upgraded to a write lock, or the write lock is downgraded to a read lock.

6. The method for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism according to claim 5, characterized in that: The lock upgrade and downgrade specifically include: When a read thread needs to perform a write operation, the read lock is upgraded to a write lock; when a write thread completes the write operation, the write lock is downgraded to a read lock to allow other read and write operations to continue.

7. A system for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism, characterized in that: include: Client side; The client sends a read / write request to any node in the Elasticsearch cluster. If the node receives the read / write request, it feeds back to the first response module. Otherwise, throw an exception; First response module; The first response module controls the node that receives the request as a coordination node, routes the read / write request to the correct data node, and feeds back to the second response module; Otherwise, throw an exception; Second response module; The second response module controls the coordination node to determine the target primary shard to be written according to the document ID and the index setting, and after the target primary shard is successfully determined, feedback is sent to the third response module; Otherwise, throw an exception; The third response module; The third response module controls the coordination node to forward the read / write request to the data node where the primary shard is located. After receiving the read / write request, the primary shard on the data node first writes the document into the Lucene index structure in the memory and feeds back to the fourth response module. Otherwise, throw an exception; Fourth response module; After the primary shard on the data node receives the read / write request, the fourth response module controls the Lucene index to select and execute different read / write lock strategies according to the content of the read / write operation. The content of the read / write operation matches the corresponding scenario. After the read / write lock strategy is successfully executed, it is fed back to the fifth response module. Otherwise, throw an exception; Fifth response module; The fifth response module maps the words in the document to a list of documents containing the words; after the inverted index is successfully executed, the data is stored in the data node, and after success, it is fed back to the sixth response module; Otherwise, feedback is sent to the client and an exception is thrown; Sixth response module; After the inverted index is completed, the sixth response module coordination node aggregates the results from each data node, sorts and paginates them, and finally returns the results to the client.

8. An electronic device, characterized in that: The device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for improving Lucene performance in an Elasticsearch cluster based on a read-write lock mechanism according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on the electronic device, the electronic device executes the method for improving the performance of Lucene in an Elasticsearch cluster based on a read-write lock mechanism as described in any one of claims 1 to 6.

Images