Search engine index building apparatus and method, index writing system and method
By using distributed computing based on index configuration in an offline state to transform data into index building results, the single-node bottleneck problem of Elasticsearch clusters during large-scale data writing is solved, achieving efficient index building and improved data timeliness.
Patent Information
- Application Number
- CN202611390599.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-09-12
- Filing Date
- 2026-09-08
- Publication Date
- 2026-11-03
Smart Images
Figure CN123030604A_ABST
Abstract
Description
[0001] This application claims priority to Chinese Patent Application No. 202511304088.0, filed on September 12, 2025, entitled "Search Engine Index Building Apparatus and Method, Index Writing System and Method", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of data processing technology, and in particular to a search engine index building apparatus and method, and an index writing system and method. Background Technology
[0003] Driven by the trend of digital transformation, enterprises are generating ever-increasing amounts of data in their daily operations and management. To meet the needs of rapid business queries and retrieval, more and more enterprises are choosing to write their data to Elasticsearch (ES, an open-source distributed search and analytics engine) for centralized storage and efficient searching. This data management approach can significantly improve information retrieval efficiency and support business analysis and decision-making across multiple scenarios.
[0004] Elasticsearch is a general-purpose search engine. Existing solutions provide a stateless HTTP (Hypertext Transfer Protocol) API (Application Programming Interface) for writing data to the cluster. Data is first sent to a coordinating node, which then distributes the data to the Primary and Replica nodes of the index shards based on the cluster's metadata. Existing solutions improve write concurrency by adding coordinating nodes, but they do not address the bottleneck of single-node processing capacity and cannot perform offline builds. Existing solutions provide a resume-from-breakpoint solution, but their data processing efficiency is limited. Current data processing primarily utilizes Elasticsearch's batch write API for storage operations. However, in the current environment, writing large amounts of data via the batch interface results in long task execution times, leading to significant data ingestion delays and severely impacting data timeliness.
[0005] During large-scale data writes, the Elasticsearch cluster experiences immense write pressure, especially during prolonged, continuous batch operations, leading to a significant performance degradation. This directly results in numerous online user queries timing out or becoming unavailable, severely impacting the query experience. Summary of the Invention
[0006] This application provides a search engine index building apparatus and method, and an index writing system and method to solve one or more of the above-mentioned technical problems.
[0007] In a first aspect, embodiments of this application provide a search engine index building apparatus, comprising: an acquisition unit, configured to acquire the index configuration of an online search engine, and determine the number of shards and the data storage method based on the index configuration; a partitioning unit, configured to acquire data to be written, and group the data to be written according to the number of shards to obtain partition data; and a building unit corresponding one-to-one with the partition data, wherein the building unit is configured to: write the corresponding partition data according to the data storage method to obtain an index building result.
[0008] Secondly, embodiments of this application provide an index writing system, including a building end, a management end, and a loading end. The building end is communicatively connected to the management end and includes the aforementioned search engine index building device. The loading end is configured with a loading plugin. The building end is used to upload the index building results of the search engine index building device to a distributed file system in an offline state, and send an upload notification to the management module after completing the upload. The management end is used to call the loading plugin and submit the file path information of the index building results when it determines that the received upload notification meets preset conditions, so that the loading end writes the index building results to the search engine cluster according to the file path information.
[0009] Thirdly, embodiments of this application provide a method for constructing a search engine index, comprising: an acquisition unit acquiring the index configuration of an online search engine, and determining the number of shards and the data storage method based on the index configuration; a partitioning unit acquiring data to be written, and grouping the data to be written according to the number of shards to obtain partition data; and a construction unit writing the corresponding partition data according to the data storage method to obtain an index construction result; wherein the construction unit corresponds one-to-one with the partition data.
[0010] Fourthly, this application provides an index writing method applied to the aforementioned index writing system. The method includes: when the building end is offline, it uploads the index building result of the search engine index building device to a distributed file system, and sends an upload notification to the management module after completing the upload; when the management end determines that the received upload notification meets preset conditions, it calls the loading plugin of the loading end and submits the file path information of the index building result, so that the loading end writes the index building result to the search engine cluster according to the file path information.
[0011] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements the above-described method when executing the computer program.
[0012] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.
[0013] Compared with related technologies, this application has the following advantages:
[0014] This application provides a search engine index building apparatus and method, and an index writing system and method. According to the embodiments of this application, data to be written can be converted into index building results offline, based on the search engine's index configuration. This conversion operation is implemented through distributed offline computing, improving the efficiency of index building. The index configuration of the index building result is the same as that of the search engine's index configuration; therefore, it can be used to import into the index of the search engine cluster, reducing index writing time, improving data timeliness, and alleviating the pressure of writing operations on the online search engine cluster.
[0015] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description
[0016] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.
[0017] Figure 1 This paper shows a structural block diagram of a search engine index building device provided in an embodiment of the present application;
[0018] Figure 2 This paper shows a structural block diagram of an index writing system provided in an embodiment of this application;
[0019] Figure 3 This illustration shows a data processing flow diagram of an index writing system provided in an embodiment of this application;
[0020] Figure 4 This paper illustrates a data flow diagram of an index writing system provided in an embodiment of this application.
[0021] Figure 5 This illustration shows a data processing logic diagram for a scheduling exception provided in an embodiment of this application;
[0022] Figure 6 A flowchart of a search engine index construction method provided in an embodiment of this application is shown;
[0023] Figure 7 A flowchart of an index writing method provided in an embodiment of this application is shown;
[0024] Figure 8 A block diagram of an electronic device used to implement embodiments of this application is shown. Detailed Implementation
[0025] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.
[0026] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.
[0027] Elasticsearch (ES) provides a native write method—sending data to the ES cluster via the HTTP API. However, this native API write method has a long chain and is prone to a long-tail effect (a few slow nodes slow down the overall speed), making it more suitable for writing small amounts of data in real-time. For offline data warehouses that need to inflate large amounts of data in a short period, Elasticsearch's native API easily exposes the following problems: First, long write times affect data timeliness; second, the online ES cluster experiences a surge in pressure, with cluster load exceeding 85% during writes, leading to a 300% increase in query response time, causing online business queries to stall or even time out, impacting the performance of the online cluster.
[0028] Based on this, this application provides a search engine index building apparatus and method, and an index writing system and method. Based on this application, the chain length can be reduced by using a striped approach for local writing. By building the index offline and then loading it into the online cluster, data can be written to the Elasticsearch cluster efficiently while achieving isolation. In the above scenario, for over 200 million data points (approximately 20TB in size), the overall chain time for offline building and loading is reduced from 18 hours using the existing API solution to 7 hours. Because offline building is used, there is no pressure on the online Elasticsearch cluster during the building process, and only a small amount of disk copy IO consumption occurs during loading, with no noticeable impact on the user's query experience. This application balances the speed of index writing to the cluster and the stability of the cluster.
[0029] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0030] This application provides a search engine index building apparatus. For example... Figure 1 The diagram shown is a structural block diagram of a search engine index building apparatus 100 according to an embodiment of this application. The apparatus may include:
[0031] The acquisition unit 101 is used to acquire the index configuration of the online search engine and determine the number of shards and the data storage method based on the index configuration; the partitioning unit 102 is used to acquire the data to be written and group the data to be written according to the number of shards to obtain partition data; the construction unit 103 corresponds one-to-one with the partition data, and the construction unit is used to write the corresponding partition data according to the data storage method to obtain the index construction result.
[0032] In this embodiment, distributed computing can be used for data processing. The search engine's index configuration includes index-level metadata and field-level schema definitions. Index-level metadata includes, but is not limited to, the index's storage structure, number of shards, replicas, and operational parameters such as the tokenizer. Field-level schema definitions determine the structure and interpretation rules of the data within the index, such as the storage method, type, and search method for each field. Field-level schema definitions include, but are not limited to, field types, analysis methods, storage options, field sub-attributes, and nested types.
[0033] In this embodiment, the search engine is one that needs to be written with batch data and can exist in the form of a search engine cluster. Obtaining the index configuration of the online search engine ensures that the final generated index construction result meets the requirements of the online search engine for index files, thereby avoiding phenomena such as loading failures or query errors. Based on the obtained index configuration, the number of shards and the data storage method are determined. The number of shards is the number of primary shards contained in each index, determined based on index-level metadata. The data storage method is determined based on the field-level schema definition (such as the tokenizer type). For example, field types include, but are not limited to: text type (for full-text search) and keyword type (for exact matching, aggregation, and sorting).
[0034] The data to be written is batch data used to write to the online search engine. This data can be obtained from a data warehouse, including but not limited to a Hive data warehouse. The data to be written is grouped according to the number of shards, resulting in a number of partition data groups equal to the number of shards. Each shard is then assigned to a building unit for processing, meaning there is a one-to-one correspondence between partition data and building units. In one possible implementation, the size range of the partition data can be set according to actual needs, for example, it may include, but is not limited to, 40-60GB. The search engine index building device includes multiple processing nodes, i.e., multiple building units. Multiple building units can process data in parallel, shortening the data processing chain and reducing the waiting time between shards during serial shard writing, thus improving index building efficiency. Each building unit processes the corresponding partition data according to the data storage method, and then stores the processing results to obtain the index building result.
[0035] This application provides a search engine index building apparatus. According to embodiments of this application, data to be written can be converted into index building results offline, based on the search engine's index configuration. This conversion operation is implemented through distributed offline computing, improving the efficiency of index building. The index configuration of the index building result is the same as that of the search engine; therefore, it can be imported into the index of the search engine cluster, reducing index writing time, improving data timeliness, and alleviating the pressure of write operations on the online search engine cluster.
[0036] In one possible implementation, the partitioning unit is specifically used for: calculating the primary key hash value of the data to be written; and allocating the data to be written to different partitions according to the primary key hash value, thereby obtaining multiple partition data.
[0037] In this embodiment, a primary key is determined in the data to be written. The hash value of the primary key is calculated using the same hash algorithm as the search engine. The hash value is then multiplied by the number of shards to determine which shard the data belongs to, thus allocating the data to be written to different partitions, resulting in multiple partitioned data. In this step, each index built offline corresponds exactly to a shard in the online search engine, allowing for direct processing during loading. This step ensures even data distribution and avoids the long-tail effect.
[0038] In one possible implementation, the building unit is configured to: start a single-instance search engine node, create a temporary index in the single-instance search engine node, write corresponding partition data to the temporary index, refresh the temporary index, and shut down the single-instance search engine node.
[0039] In this embodiment, a single-instance search engine node can be locally started on a compute node. This search engine node runs independently of the online search engine cluster and is used to perform index building operations in an offline environment. Within this single-instance search engine node, a temporary index with the same structure as the target shard is created based on pre-acquired target index configuration information (including the number of index shards, data storage method, etc.). Data allocated to the current partition is written in batches to the temporary index to generate the underlying index file of the corresponding shard. After the data writing is completed, a refresh operation is performed on the temporary index to persist the data in the memory buffer and generate a complete underlying index segment file, thus obtaining the index building result. After the refresh is completed, the single-instance search engine node is shut down to release computing and storage resources.
[0040] In this possible implementation, in a search engine (e.g., a Lucene-based search engine), index data is typically first stored in a memory buffer and a transaction log, before a physical index segment file is formed for retrieval. A refresh operation is a process of writing data from the memory buffer to disk and generating one or more new index segment files. This operation updates the segment metadata of the index, making the newly written data retrieveable in subsequent search requests. Once the refresh is complete, the data to be committed in memory is converted into persistent physical files, and these files are formatted to match the underlying index structure (e.g., the Lucene index) used by the online search engine cluster. In this embodiment, by performing a refresh after all batch writes are complete (rather than relying on the default automatic refresh mechanism), it is ensured that a complete and intact underlying index segment file is generated in one go. This allows the generated segment files to be directly migrated to the target search engine cluster without repeating the index building process, saving deployment time and reducing the impact on the online cluster's write performance.
[0041] In one possible implementation, the building unit is configured to: in response to receiving a write request, write multiple data entries from the partition data in parallel into the corresponding temporary index.
[0042] In this embodiment, when the construction unit receives a write request, it first identifies the partition data associated with the request and its corresponding temporary index according to a preset data sharding rule. Then, the construction unit treats the partition data as a batch processing unit, using multi-threading or multi-tasking to write multiple records into the temporary index simultaneously. In an optional implementation, parallel writing can be achieved by submitting multiple records to the temporary index in a single instruction using a batch write interface (such as the Bulk API) to complete persistence, reducing the overhead of multiple network interactions and shortening the overall write latency.
[0043] This application provides a search engine index building apparatus, comprising: an acquisition unit for acquiring the index configuration of an online search engine and determining the number of shards and data storage method based on the index configuration; a partitioning unit for acquiring data to be written and grouping the data to be written according to the number of shards to obtain partition data; and building units corresponding one-to-one with the partition data, each building unit being used to: write the corresponding partition data according to the data storage method to obtain an index building result. According to an embodiment of this application, data to be written can be converted into an index building result offline, based on the search engine's index configuration. This conversion operation is implemented through distributed offline computing, improving the efficiency of index building. The index configuration of the index building result is the same as that of the search engine's index configuration; therefore, it can be used to import into the index of a search engine cluster, reducing index writing time, improving data timeliness, and alleviating the pressure of writing operations on the online search engine cluster.
[0044] This application also provides an index writing system, see [link to relevant documentation]. Figure 2 The diagram shows the structure of the index writing system. The system includes a build terminal 10, a management terminal 20, and a loading terminal 30. The build terminal 10 is communicatively connected to the management terminal 20 and includes the aforementioned search engine index building device 100. The loading terminal 30 is configured with a loading plugin 300. The build terminal 10 is used to upload the index building results of the search engine index building device to a distributed file system in offline mode, and sends an upload notification to the management module after completing the upload. The management terminal 20 is used to, when determining that the received upload notification meets preset conditions, call the loading plugin 300 to submit the file path information of the index building results, so that the loading terminal 30 writes the index building results to the search engine cluster according to the file path information.
[0045] In this embodiment, the build terminal, based on the aforementioned search engine index building device, obtains the index building result offline and uploads it to the distributed file system. The build terminal communicates with the management module and sends an upload notification to the management module after the upload is complete. Preset conditions can be set according to actual needs to control the successful transmission of the index building result to the distributed file system. When the management terminal determines that the received upload notification meets the preset conditions, it calls the loading plugin (e.g., the ES plugin when the search engine is ES), submitting the file path information of the index building result, so that the loading terminal writes the index building result to the search engine cluster according to the file path information.
[0046] In this embodiment, the build end can complete the target index construction operation in an offline environment logically isolated from the online search engine cluster based on the aforementioned search engine index building device, obtaining the corresponding index construction result. The index construction result includes a set of underlying index files (e.g., Lucene index files) corresponding to the target index shard structure. This set of index files can be directly loaded into the target search engine cluster without repeated construction. After construction is completed, the build end uploads the index construction result to a distributed file system (e.g., HDFS file system). The distributed file system provides persistent data storage and distributed access capabilities, ensuring high availability and parallel download capability of the index files. The build end maintains a communication connection with the management end (management module). After successfully uploading the index files, it sends a notification message to the management end indicating that the upload is complete. In one optional implementation, before sending the notification information, the building end can confirm that the index building results have been completely and losslessly transmitted to the distributed file system based on pre-set verification conditions (such as file integrity verification, file quantity / size comparison, upload time threshold, etc.). These pre-set conditions can be flexibly configured according to actual business needs to ensure the quality and consistency of transmitted data, thereby avoiding abnormal operation of the search engine cluster due to incomplete file loading. For example, the pre-set condition could be: the index building results corresponding to all shards have been successfully uploaded and verified.
[0047] Specifically, in one possible implementation, the management terminal determines that the index construction result verification has passed, which may include: the index construction results corresponding to each shard in the distributed file system exist, and the file size is consistent with the metadata reported by the construction terminal; the CRC checksum of each index construction result matches the checksum recorded when the construction terminal uploaded it; and the total number of shards in the index construction results is consistent with the number of shards in the index configuration. When the management terminal determines that all of the above conditions are met, it confirms that the preset condition is valid.
[0048] When the management terminal determines that the received upload notification meets the preset conditions, it will call the loading plugin pre-deployed on the loading terminal to submit the file path information of the index building result in the distributed file system to the loading terminal. The loading terminal obtains the corresponding index file from the distributed file system based on the file path information and loads it directly into the corresponding shard of the target search engine cluster. This achieves rapid deployment without having to re-execute the index building process in the production environment, shortens the time for new indexes to go online, and reduces the write pressure on the online cluster.
[0049] This application provides an index writing system. According to this embodiment, data to be written can be converted into index building results offline, based on the search engine's index configuration. This conversion operation is implemented through distributed offline computing, improving the efficiency of index building. The index configuration of the index building result is the same as that of the search engine; therefore, it can be imported into the index of the search engine cluster, reducing index writing time, improving data timeliness, and alleviating the pressure on the online search engine cluster from the writing operation.
[0050] In one possible implementation, the management terminal is also used to: send an alarm message to the construction terminal when any index construction result of the data to be written fails to be uploaded.
[0051] In this possible implementation, when the management terminal receives and monitors the index building results uploaded to the distributed file system by the build terminal, if it detects that the index building file corresponding to any shard has not been transmitted within a preset time window, or that a mismatch with preset integrity verification rules occurs during transmission, it can determine that the index building result upload has failed. After determining the upload failure, the management terminal immediately sends an alarm message to the build terminal to which the task belongs. The alarm message includes the shard identifier, the failure reason identifier, and the build task identifier, etc., to guide the build terminal to perform data reconstruction or retransmission operations for that shard. In this way, real-time anomaly detection and source-end alarms can be achieved in the index building result transmission stage, improving the robustness and operational efficiency of the entire offline build and loading system.
[0052] In one possible implementation, upon receiving an alarm, the build server can automatically re-execute the upload operation for failed index building results according to a preset retry strategy. This could include setting a maximum number of retries and a retry interval. Upon successful retry, a new upload notification is sent to the management server. If the upload still fails after reaching the maximum number of retries, the build server saves the index building result to the local disk and marks it as abnormal. This information, combined with the processed fragment identifiers, file offsets, and CRC checksums recorded in the aforementioned breakpoint resume mechanism, is used for subsequent manual intervention or task recovery. This retry handling method improves the system's fault tolerance during the index building result upload phase, preventing the interruption of the overall offline build and loading task due to the failure of individual fragment uploads.
[0053] See Figure 5 The diagram illustrating the data processing logic in case of scheduling anomalies demonstrates a specific implementation method for sending alarm information to the build end. A task latency threshold can be preset. When it exceeds 30%, measures can be taken to increase the task priority, specifically by allocating GPU resources through the resource scheduler. When it exceeds 50%, measures can be taken to request additional containers from YARN (YetAnother Resource Negotiator), specifically by dynamically expanding Spark Executors. When it exceeds 120%, measures can be taken to terminate the task and send an SMS alarm, specifically by generating a failure analysis report.
[0054] See Figure 3 The diagram illustrates the data processing flow of the index writing system. It shows how data (DATA) in the Hive database is processed by a single Elasticsearch instance on the build end to obtain the index building result, how the processing status and other information are reported to the management end, and how the index building result (Lucene) is uploaded to the distributed file system HDFS. The diagram also shows an example of the management end loading the Elasticsearch cluster loading plugin, submitting the file path information of the index building result, and the loading end writing the index building result to the Elasticsearch cluster based on the file path information obtained from the loading plugin. Combined with... Figure 4 The diagram shown illustrates the data flow of the index writing system. The following is a specific embodiment illustrating the data processing of this system.
[0055] Based on the aforementioned processing flow between the build end, management end, and loading end, it can be seen that... Figure 4The data flow diagram shown is as follows: The index building results (i.e., Lucene index files) generated by the build end in offline mode, corresponding one-to-one with the target index shards, are uploaded by the build end to the distributed file system HDFS, and an upload notification is sent to the management end after the upload is completed; after the management end determines that the received upload notification meets the preset conditions (e.g., the index building results corresponding to all shards have been successfully uploaded and verified), it calls the loading plugin pre-deployed by the loading end to submit the file path information of the index building results in HDFS to the loading end; the loading end downloads the corresponding Lucene file from HDFS according to the received file path information, and writes it in parallel to the corresponding shard of the target search engine cluster through zero-copy loading, thus forming a complete data flow path from the build end through HDFS to the loading end, and then to the target search engine cluster.
[0056] The build phase primarily handles index building. The search engine can be Elasticsearch (ES), and the build phase uses Spark (an open-source distributed computing framework for parallel processing of large-scale data) as its index building mechanism. In the Job phase, it reads the Setting information (as index-level metadata) and Mapping information (as field-level schema definitions) from the online Elasticsearch database to be written to the target index. In the Task phase, Spark reads data from the data warehouse Hive and distributes the data across different partitions based on the primary key hash value, with the number of partitions equal to the number of shards specified in the index Setting information. When processing partitioned data, a single-instance Elasticsearch Node is first created. Then, based on the previously obtained Setting and Mapping information, a single-shard temporary index is created, and the partitioned data is written to the temporary index using a Bulk method. After writing, the index is refreshed, and the Elasticsearch Node instance is shut down. The resulting Lucene file (i.e., the index building result) is uploaded to the HDFS (Hadoop Distributed File System). Simultaneously, the build phase periodically reports the task execution status to the management end, including task status, execution progress, and build rate.
[0057] It should be noted that in this possible implementation, a Job represents the complete execution chain from data reading to final output. Spark splits a Job into multiple Stages based on data dependencies (narrow or wide dependencies). A Stage is a phase of physical execution, and Tasks can be executed in parallel within a Stage. A Stage is further broken down into multiple Tasks, each processing a data partition. In this embodiment, data from the same partition is grouped into a "strip" and given to the same Spark Task to build a single-partition Lucene file.
[0058] The management console primarily handles the observation and display of the entire execution process, provides task anomaly alerts, and manages tasks such as scheduling, reloading, and index rollback of loading tasks. Upon receiving build information from the build console, the management console creates a task entry and waits for all shard build tasks to complete. Then, it calls the Elasticsearch loading plugin API, submitting the Lucene file path information to start the loading task. The management console periodically retrieves loading progress information, notifying the user of task completion upon completion and issuing alerts if the task fails.
[0059] The loading module is primarily responsible for downloading and loading index files. It uses the Elasticsearch loading plugin to download Lucene files from the HDFS distributed file system to the local machine, and then calls the underlying API to load the Lucene files into the index. The loading plugin supports zero-copy loading, directly mapping the downloaded Lucene files to the target index structure without additional file copying. The loading concurrency can be controlled to not exceed twice the number of data nodes in the target cluster, balancing loading efficiency with cluster resource consumption.
[0060] In this embodiment, Spark distributed computing technology is introduced to fully utilize offline distributed computing and storage resources to generate Lucene files on various nodes within the YARN cluster. Then, an Elasticsearch plugin loads the Lucene files into the Elasticsearch cluster to achieve rapid offline construction with high concurrency. Specifically, in the Mapper stage (responsible for data partitioning), the data is partitioned according to Elasticsearch sharding rules, and the data is distributed to the Reducer stage (responsible for actually building the sharded index). Each Reducer Task simulates an Elasticsearch environment to access local data and ultimately generate Lucene files. These Lucene files are then uploaded to the HDFS distributed file system for storage, and the Elasticsearch loading plugin is called to download the Lucene files and load them into the cluster's index.
[0061] In this embodiment, the following are achieved: improved write efficiency, with write speed increased by 2.6 times using Spark distributed computing; cluster isolation protection, with offline environment construction avoiding impact on production clusters; improved fault tolerance mechanism, supporting task retries and breakpoint resumption, and supporting loading onto multiple clusters to achieve high availability; and a management platform that supports tasks, scheduling monitoring, task cancellation, modifying rate limits, index rollback, and other operations, and supports viewing observable indicators such as task progress and execution rate, as well as monitoring and alarm functions.
[0062] The embodiments of this application were validated with a 20TB data scale. Compared with traditional solutions, the construction efficiency of this application is improved by 2.6 times and the cluster load is reduced by 53%. Specific experimental data can be found in Table 1.
[0063] index Traditional API solutions This application proposal Increase Single-node write throughput 50MB / s 200MB / s 300% Overall build time 18 hours 7 hours 61% Cluster CPU utilization peak 89% 42% 53%
[0064] Table 1
[0065] In one possible implementation, the specific functions of each module of the system can be configured according to actual needs. For example, a distributed offline index building system includes: a data access module for reading raw data from a data source and partitioning it according to sharding rules; a local building unit configured with a single-instance Elasticsearch node for generating Lucene files; a distributed storage module for storing the generated Lucene files; an index loader for loading Lucene files from distributed storage to the target cluster; and a monitoring and scheduling center for coordinating the work of each module and providing feedback on task status. The data access module includes: a data source connector supporting Hive / Spark data source access; and a sharding strategy engine for data distribution based on Elasticsearch routing algorithms. The local building unit includes: a temporary index generator for simulating an Elasticsearch environment to build temporary indexes; and a file encapsulation component for packaging index data into Lucene format.
[0066] In its implementation, the monitoring and scheduling center adopts a Prometheus (Prometheus Monitoring System and Time Series Database) + Grafana (Grafana visualization tool) architecture, collecting Elasticsearch node metrics via the JMX (Java Management Extensions) protocol. When a shard build delay exceeds a threshold, the following responses are automatically triggered: the task priority is increased to the highest level; additional computing resources are dynamically requested from YARN; and operations personnel are notified to intervene via Kafka (an open-source distributed event streaming platform). The data access module supports multiple data format conversions and has built-in Avro (Apache Avro data serialization system), Parquet (Apache Parquet columnar storage format), and ORC (Optimized Row Columnar) file parsers, which can automatically identify data patterns and generate corresponding mappings. Specifically, for sensitive data in the financial industry, an AES-256 (Advanced Encryption Standard) encrypted transmission channel is integrated to ensure data confidentiality during transmission. Local building block definition: An isolated environment running within Spark Executors, containing lightweight Elasticsearch node instances (1GB memory limit) used to generate Lucene files without impacting the production cluster. Sharding strategy engine: Based on the following algorithm formula: N = min(ceil(data_size / 50GB), cluster_node_count*3). Constraints: Single shard size range [40GB, 60GB]. Dynamic scaling scheduler, working mechanism: Automatically adjusts the number of SparkExecutors based on YARN queue resource levels, scaling formula: executors = max(4, ceil(pending_tasks / 3)). Index loader, key features: Supports zero-copy loading (via mmap mapping of Lucene files), parallelism control: Loading concurrency ≤ target cluster data node count × 2. The breakpoint resume mechanism is implemented by storing a checkpoint file (checkpoint.json) in HDFS, which records: the processed fragment ID, file offset, and CRC (Cyclic Redundancy Check) value.In one possible implementation, Lucene file pre-generation is performed by the local building unit, dynamic sharding calculation is performed by the sharding strategy engine, cluster-agnostic loading is performed by the index loader, and elastic resource scheduling is performed by the auto-scaling scheduler.
[0067] In one possible implementation, the distributed offline index building method includes the following steps: S1: Read source data through a Spark job and partition it according to a preset number of shards; S2: Start an independent Elasticsearch node instance in each Spark partition task; S3: Write the partition data to a local Lucene file using the Bulk API; S4: Upload the generated Lucene file to a distributed file system; S5: Load the Lucene file to the target cluster in parallel using an index loading plugin. In step S2: Each Spark Task creates an independent NodeClient instance; a custom ClusterName is set to avoid cluster interference; and temporary node processes are automatically cleaned up. After step S4, the method also includes: S6: The monitoring center collects the task progress of each node; S7: When a task failure is detected, a retry mechanism is triggered; S8: After loading is completed, an index health check is performed.
[0068] In one possible implementation, a data processing system is provided, integrating the aforementioned index building system, and including: a data warehouse module, a real-time query cluster, and an offline building cluster. The offline building cluster is deployed with a YARN resource manager, HDFS distributed storage, and an automatic scaling scheduler.
[0069] In one possible implementation, data processing can be performed according to the following steps:
[0070] def offline index building process():
[0071] # Initialization phase
[0072] 1. The monitoring and scheduling center receives the build task and parses the setting / mapping of the target index;
[0073] 2. The sharding strategy engine calculates the optimal number of shards N (formula: N=min(ceil(data_size / 50GB),cluster_node_count*3)).
[0074] # Data Building Phase
[0075] 3. Spark job execution:
[0076] 3.1 The data source connector reads source data from Hive;
[0077] 3.2 Perform hash partitioning based on the number of shards N;
[0078] 3.3 Within each Spark Task:
[0079] a) Start a single-instance Elasticsearch node (set an independent ClusterName);
[0080] b) The temporary index generator creates a single-shard temporary index;
[0081] c) Write partition data via BulkAPI (rate limit: 200MB / s);
[0082] d) The file encapsulation component generates Lucene files;
[0083] e) Upload the Lucene file to HDFS (with CRC32 checksum).
[0084] # Exception handling process
[0085] 4. Build failure handling:
[0086] 4.1 Node crash: Automatically clean up temporary processes and retry the task (up to 3 times).
[0087] 4.2 HDFS write failure: Record the failed fragment ID and save it to the local disk for manual recovery;
[0088] 4.3 Data format error: Skip the exception record and log the error, triggering a Kafka alert.
[0089] # Index loading phase
[0090] 5. Index loaders execute in parallel:
[0091] 5.1 Download the Lucene file from HDFS to the target cluster data node;
[0092] 5.2 Call the ES underlying Segment loading API;
[0093] 5.3 Health check: Verify fragment status, copy allocation, and document count.
[0094] # Post-processing
[0095] 6. After successful loading: Delete temporary HDFS files and update the metadata database;
[0096] 7. Failure Rollback: Activate the previous version index and mark the current task as obsolete.
[0097] This application embodiment processes data in shards to reduce the pressure on a single node, generates Lucene files in advance to reduce real-time write load, and ensures data reliability through a replication mechanism.
[0098] Corresponding to the application scenario of the apparatus provided in the embodiments of this application, the embodiments of this application also provide a method for constructing a search engine index. For example... Figure 6 The diagram shown is a flowchart of a search engine index construction method according to an embodiment of this application. The method may include:
[0099] See Figure 6 The method includes: step S601, the acquisition unit acquires the index configuration of the online search engine, and determines the number of shards and the data storage method based on the index configuration; step S602, the partitioning unit acquires the data to be written, and groups the data to be written according to the number of shards to obtain partition data; step S603, the construction unit writes the corresponding partition data according to the data storage method to obtain the index construction result; the construction unit corresponds one-to-one with the partition data.
[0100] According to the embodiments of this application, data to be written can be converted into index building results offline, based on the search engine's index configuration. This conversion operation is implemented through distributed offline computing, improving the efficiency of index building. The index configuration of the index building result is the same as that of the search engine; therefore, it can be imported into the index of the search engine cluster, reducing index writing time, improving data timeliness, and alleviating the pressure of write operations on the online search engine cluster.
[0101] In one possible implementation, the data to be written is grouped according to the number of shards to obtain partitioned data, including: calculating the primary key hash value of the data to be written; and allocating the data to be written to different partitions according to the primary key hash value to obtain multiple partitioned data.
[0102] In one possible implementation, writing the corresponding partition data according to the data storage method includes: starting a single-instance search engine node, creating a temporary index in the single-instance search engine node, writing the corresponding partition data to the temporary index, refreshing the temporary index, and closing the single-instance search engine node.
[0103] In one possible implementation, writing the corresponding partition data to the temporary index includes: in response to receiving a write request, writing multiple data entries from the partition data into the corresponding temporary index in parallel.
[0104] Corresponding to the application scenario of the system provided in the embodiments of this application, the embodiments of this application also provide an index writing method. For example... Figure 7The diagram shown is a flowchart of an index writing method according to an embodiment of this application. The method may include:
[0105] See Figure 7 The method includes: Step S701, in an offline state, the building end uploads the index building results of the search engine index building device to a distributed file system, and sends an upload notification to the management module after completing the upload. Step S702, when the management end determines that the received upload notification meets preset conditions, it calls the loading plugin of the loading end and submits the file path information of the index building results, so that the loading end writes the index building results to the search engine cluster according to the file path information.
[0106] According to the embodiments of this application, data to be written can be converted into index building results offline, based on the search engine's index configuration. This conversion operation is implemented through distributed offline computing, improving the efficiency of index building. The index configuration of the index building result is the same as that of the search engine; therefore, it can be imported into the index of the search engine cluster, reducing index writing time, improving data timeliness, and alleviating the pressure of write operations on the online search engine cluster.
[0107] In one possible implementation, the method may further perform the following steps: when the upload of any index construction result of the data to be written fails, send an alarm message to the construction end.
[0108] The effects of each step in the methods of the embodiments of this application can be found in the corresponding descriptions of the above-mentioned apparatus or system, and they have corresponding beneficial effects, which will not be repeated here.
[0109] Figure 8 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 8 As shown, the electronic device includes a memory 801 and a processor 802. The memory 801 stores a computer program that can run on the processor 802. When the processor 802 executes the computer program, it implements the method described in the above embodiments. The number of memories 801 and processors 802 can be one or more.
[0110] The electronic device also includes:
[0111] The communication interface 803 is used to communicate with external devices and exchange and transmit data.
[0112] If the memory 801, processor 802, and communication interface 803 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0113] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.
[0114] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.
[0115] This application provides a computer program product, wherein the computer program product includes a computer program, which, when executed by a processor, implements the method provided in this application embodiment.
[0116] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.
[0117] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.
[0118] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.
[0119] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).
[0120] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.
[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0122] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.
[0123] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.
[0124] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.
[0125] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).
[0126] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.
[0127] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.
[0128] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A search engine index building apparatus, comprising: The acquisition unit is used to acquire the index configuration of the online search engine and determine the number of shards and the data storage method based on the index configuration. A partitioning unit is used to acquire data to be written, and to group the data to be written according to the number of fragments to obtain partitioned data; The construction units, which correspond one-to-one with the partition data, are respectively used to: write the corresponding partition data according to the data storage method to obtain the index construction result.
2. The search engine index building apparatus according to claim 1, wherein, The partitioning unit is specifically used for: Calculate the primary key hash value of the data to be written; Based on the primary key hash value, the data to be written is allocated to different partitions, resulting in multiple partition data.
3. The search engine index building apparatus according to claim 1, wherein, The building unit is used for: Start a single-instance search engine node, create a temporary index in the single-instance search engine node, and write the corresponding partition data into the temporary index; Refresh the temporary index and close the single-instance search engine node.
4. The search engine index building apparatus according to claim 3, wherein, The building unit is used for: In response to receiving a write request, multiple data entries from the partition data are written in parallel to the corresponding temporary index.
5. An index writing system, comprising a building end, a management end, and a search loading end, wherein the building end is communicatively connected to the management end, and the building end comprises the search engine index building device according to any one of claims 1-4; the loading end is configured with a loading plugin; The building end is used to upload the index building results of the search engine index building device to the distributed file system in an offline state, and send an upload notification to the management module after the upload is completed. The management terminal is used to determine that the received upload notification meets preset conditions, and then call the loading plugin to submit the file path information of the index building result, so that the loading terminal writes the index building result to the search engine cluster according to the file path information.
6. The index writing system according to claim 5, wherein, The management terminal is also used for: If any index construction result of the data to be written fails to be uploaded, an alarm message is sent to the construction end.
7. A method for constructing a search engine index, comprising: The acquisition unit acquires the index configuration of the online search engine and determines the number of shards and the data storage method based on the index configuration; The partition unit obtains the data to be written, and groups the data to be written according to the number of fragments to obtain partition data; The building unit writes the corresponding partition data according to the data storage method to obtain the index building result; the building unit corresponds one-to-one with the partition data.
8. An index writing method, applied to the index writing system according to any one of claims 5-6, the method comprising: When the construction end is offline, it uploads the index construction results of the search engine index building device to the distributed file system, and sends an upload notification to the management module after the upload is completed. When the management terminal determines that the received upload notification meets the preset conditions, it calls the loading plugin of the loading terminal and submits the file path information of the index building result, so that the loading terminal writes the index building result to the search engine cluster according to the file path information.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of claim 7.
10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of claim 7.