Data writing method, device, storage medium and equipment

Abstract

The present application discloses a data writing method, device, storage medium and equipment. The method does not write data directly to a search engine, but is implemented by controlling the search engine to close the index, copy the data and open the index, so as to copy the shard data written in advance on the distributed file system through the inverted index to the search engine. Since the shard data written in advance on the distributed file system in an offline manner is written by the search engine's data writing tool, the data content and format are the same as those generated by writing data to the search engine online, and therefore it is equivalent to writing the shard data to the search engine. The speed of controlling the search engine to copy the shard data written in advance on the distributed file system to the corresponding folder of the search engine is much faster than writing remote big data to the search engine through a software interface. Therefore, the time to build a database on the search engine is greatly shortened.

Description

Technical Field

[0001] The present application relates to the field of big data processing, and in particular to a data writing method, device, storage medium and equipment. Background Art

[0002] Search engines, such as Elasticsearch, are currently popular distributed, highly scalable, and highly real-time search and data analysis engines that can search, analyze, and explore large amounts of existing data in a very convenient way.

[0003] However, search engines are currently unable to withstand requests for writing a large amount of data at the same time, which results in a very time-consuming process of building a database in the search engine each time. In addition, since writing data to the search engine requires frequent and large-scale IO accesses to the disk of the machine where the search engine is located, it will also cause a sharp increase in the load on the machine where the search engine is located, and even affect the search engine's indexing speed. Summary of the invention

[0004] In order to solve the above-mentioned problems existing in deploying different services in multiple different small clusters, the embodiments of the present application creatively provide a data writing method, apparatus, storage medium and device method, apparatus and computer-readable storage medium.

[0005] According to a first aspect of an embodiment of the present application, a data writing method is provided, the method comprising: in response to a request to write data, performing the following operations: controlling a search engine to close an index corresponding to the data to be written, the index being used to index shard data in a plurality of shards; controlling a search engine to copy the shard data corresponding to the shard of the index in a distributed file system to a folder corresponding to the corresponding shard of the index, the shard data being written to the distributed file system in advance according to the data to be written and the shards of the index by a data writing tool of the search engine; and controlling a search engine to open the index.

[0006] According to an embodiment of the present application, the method further includes: controlling the search engine to create an index corresponding to the data to be written; obtaining shard information corresponding to the index, the shard information including the number of shards, and a mapping relationship between the shard data and the number of shards.

[0007] According to an embodiment of the present application, the method further includes: determining a data writing tool; and writing the data to be written into the distributed file system through the data writing tool to form shard data corresponding to the shard of the index.

[0008] According to an embodiment of the present application, the data to be written is written into a distributed file system through a data writing tool to form shard data corresponding to the shard of the index, including: determining multiple first sub-data in the data to be written and shard information corresponding to each first sub-data; writing each first sub-data into the corresponding shard through the data writing tool to form a first segment file of the first shard; merging multiple first segment files located in the same shard to generate shard data of the shard.

[0009] According to an embodiment of the present application, the method further includes: obtaining transaction log data corresponding to the shard data, and storing the transaction log data in the shard corresponding to the shard data.

[0010] According to an embodiment of the present application, the data to be written is written to the distributed file system through a data writing tool to form shard data corresponding to the shards of the index, including: creating the same number of threads as the number of shards, and the threads write the data to be written to the distributed file system one-to-one concurrently through the data writing tool to form shard data corresponding to the shards of the index.

[0011] According to the second aspect of the embodiment of the present application, a device for writing data is also provided, which includes: an index closing module, used to control the search engine to close the index corresponding to the data to be written, the index including multiple shards; a data copying module, used to control the search engine to copy the shard data corresponding to the shard of the index in the distributed file system to the folder corresponding to the corresponding shard of the index, the shard data is written to the distributed file system in advance according to the data to be written and the index by a data writing tool; an index opening module, used to control the search engine to open the index.

[0012] According to the third aspect of the embodiment of the present application, a computer device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, any of the above-mentioned data writing methods is implemented.

[0013] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is also provided, on which a computer program is stored, and when the program is executed by a processor, any of the above-mentioned data writing methods is implemented.

[0014] This method does not write data directly to the search engine, but is implemented by controlling the search engine to close the index, copy the data, and open the index, and copying the shard data written in advance on the distributed file system through the inverted index to the search engine. Since the shard data written in advance on the distributed file system in an offline manner is written by the search engine's data writing tool, it has the same content and format as the data generated by writing data to the search engine online, so it is equivalent to writing the shard data to the search engine. The speed of controlling the search engine to copy the shard data written in advance on the distributed file system to the corresponding folder of the search engine is much faster than writing remote big data to the search engine through the software interface. Therefore, the time for building a database on the search engine is greatly shortened. It should be noted that since the shard data written in advance on the distributed file system is written through the native API of the search engine's data writing tool, it has the same content and format as the data generated by writing data directly to the search engine. Therefore, it can be used by copying, which is functionally equivalent to writing data directly to the search engine.

[0015] It should be understood that the teachings of the present application are not required to achieve all of the above beneficial effects, but specific technical solutions can achieve specific technical effects, and other embodiments of the present application can also achieve beneficial effects not mentioned above. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0017] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0018] Figure 1 A schematic diagram of the implementation flow of a data writing method according to an embodiment of the present application;

[0019] Figure 2 A schematic diagram of the corresponding relationship between data involved in another embodiment of the present application;

[0020] Figure 3 For this application Figure 2 The illustrated embodiment is a schematic diagram of a process flow for implementing a method for writing data;

[0021] Figure 4 A schematic diagram of a data writing method according to another embodiment of the present application;

[0022] Figure 5 This is a schematic diagram of the composition structure of the data writing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0024] The technical solution of the present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Figure 1 The main process of implementing the data writing method according to an embodiment of the present application is shown. Figure 1 , the method responds to a request to write data by performing the following operations:

[0026] Operation 110, controlling the search engine to close the index corresponding to the data to be written, where the index is used to index the shard data in the multiple shards;

[0027] Operation 120, controlling the search engine to copy the shard data corresponding to the shard of the index in the distributed file system to the folder corresponding to the corresponding shard of the index, wherein the shard data is preliminarily written into the distributed file system according to the data to be written and the shard of the index by the data writing tool of the search engine;

[0028] Operation 130: Control the search engine to open the index.

[0029] Controlling the search engine to perform corresponding operations can be achieved through the software interface (e.g., Restful API) provided by the search engine; it can also be achieved by remotely logging into the search engine and executing a script program pre-placed on the search engine, or by any other applicable method.

[0030] In operation 110 , the data to be written refers to data that is planned to be written by the current writing method but has not yet been written.

[0031] An index is a logical storage of a search engine, which can be understood as a database in a relational database. A search engine can store index data on one server, or on multiple servers after sharding. Each index has one or more shards. The index corresponding to the data to be written, which includes multiple shards, refers to the logical storage planned to store the data to be written, usually corresponding to a directory in the search engine, and each shard of the index corresponds to a folder under the directory.

[0032] When opening the index, it is equivalent to opening the above database and locking the database data storage space so that only the search engine can control the data storage space, such as the directory corresponding to the index and the folders under the directory, so it is impossible to perform a copy operation on the directory corresponding to the index.

[0033] Therefore, it is necessary to close the index corresponding to the data to be written so that the corresponding data storage space (for example, the directory corresponding to the index and the folder under the directory) can be unlocked, so that operations such as copying files can be performed to prepare for operation 120.

[0034] In operation 120, the distributed file system refers to a file system that stores files in a distributed cluster and uniformly manages files stored in each node in the distributed cluster, for example, DFS, GFS, Hadoop file system, etc.

[0035] The distributed file system can be accessed by search engines through the network, allowing the search engines to copy or pull data from it.

[0036] The shard data corresponding to the shard of the index refers to the entity data planned to be stored in each shard of the index. The shard data is generated by writing data through the search engine's data writing tool.

[0037] When offline, the search engine's data writing tool is used to write data to the distributed file system in advance according to the sharding of the data to be written and the index. The data content and format are the same as the data generated by writing the data to be written to the search engine when connected.

[0038] Therefore, after copying the shard data corresponding to the shard of the index in the distributed file system to the folder corresponding to the corresponding shard of the index, the same data generated by writing data to the folder where the search engine is located when connecting can be obtained.

[0039] Copy the shard data corresponding to the shard of the index in the distributed file system to the folder corresponding to the corresponding shard of the index, which can be achieved by executing a file copy command in the folder corresponding to the corresponding shard of the index.

[0040] At this time, by controlling the search engine to open the index corresponding to the data to be written through operation 130, the shard data copied to the folder corresponding to the corresponding shard can be loaded into the index.

[0041] In the embodiment of the present application, since the shard data to be written has been created in the distributed file system in an offline manner, when connecting to the search engine, it is only necessary to copy the pre-created shard data from the distributed file system to the folder corresponding to the shard corresponding to the index in the search engine. The execution speed of the file copy command is much faster than the speed of writing data in the software interface provided by the search engine, thereby greatly shortening the time to write data to the search engine and correspondingly shortening the time to build a database on the search engine.

[0042] In addition, since the file copy command only requires a small amount of IO access to the disk of the machine where the search engine is located, it can avoid frequent and large-scale IO access caused by writing data one by one to the search engine, and correspondingly reduces the load on the machine where the search engine is located, and reduces the impact on the search engine's data reading and data query operations to an extremely low level, thereby greatly saving the search engine's computing resources and improving the utilization rate of the search engine's computing resources.

[0043] It should be noted that Figure 1 The embodiment shown is only a basic embodiment of the data writing method of the present application, and implementers can also refine and expand it on this basis to obtain new embodiments.

[0044] Figure 2 The various data involved in the process of writing data to a search engine according to another embodiment of the present application and the corresponding relationship and specific flow between the various data are shown. Figure 3 Shows Figure 2 The illustrated embodiment implements the specific process of writing data to a search engine in the present application.

[0045] In the embodiment of the present application, Elasticsearch is used as the search engine and the Hadoop file system is used as the distributed file system.

[0046] refer to Figure 2 and Figure 3 The main process of writing the data to be written 20 into the search engine 21 in the embodiment of the present application includes:

[0047] Step 3010, controlling the search engine 21 to create an index corresponding to the data 20 to be written, and obtaining the index 211;

[0048] In the embodiment of the present application, the Restful API provided by Elasticsearch is used to control the creation of the index.

[0049] Among them, the index in the search engine 21 contains n shards, and accordingly, corresponds to n shard data: among them, it is planned to write sub-data 11, sub-data 12 and sub-data 13 in the data to be written 20 into the first shard data; it is planned to write sub-data 21, sub-data 22 and sub-data 23 into the second shard data; ...; it is planned to write sub-data n1, sub-data n2 and sub-data n3 into the Nth shard.

[0050] Step 3020, obtaining the shard information contained in the index;

[0051] For example, the number of shards N, the index mapping structure, etc.

[0052] Step 3030, determining a data writing tool of the search engine and establishing a connection with the distributed file system 22;

[0053] For Elasticsearch, input writing is implemented through the underlying data writing tool Lucene. Therefore, in this step, it can be implemented by creating a Lucene instance. And the sub-data (for example, sub-data 11, etc.) in the data to be written will be written to the shard data corresponding to the shard in the distributed file system 22 through the Lucene instance.

[0054] Step 3040, obtaining sub-data from the data to be written 20;

[0055] Assume that the current sub-data is sub-data 11.

[0056] Step 3050, obtaining the slice corresponding to the sub-data;

[0057] The shard number corresponding to the data is usually calculated by the following formula:

[0058] shard=hash(routing)%number_of_primary_shards

[0059] Among them, routing is the routing number corresponding to the data;

[0060] number_of_primary_shards is the number of shards.

[0061] Therefore, to obtain the shard corresponding to the sub-data, the shard corresponding to the sub-data can be calculated according to the above formula by checking the routing number and the number of shards (n) corresponding to the sub-data on the search engine 21.

[0062] Assume that through the above calculation, the fragment corresponding to the sub-data 11 is obtained as the first fragment.

[0063] Step 3060, writing the sub-data into the shard data on the distributed file system 22 through the data writing tool of the search engine to form a segment file of the shard data;

[0064] When writing data to the sharded data on the distributed file system 22, a document is constructed mainly through the mapping structure of the index and the data is written through the lucene native API indexWriter.addDocuments(index.parsedDoc().docs()).

[0065] Taking sub-data 11 as an example, a segment file of the first shard data will be formed in the distributed file system 22.

[0066] Step 3070, determine whether there is sub-data to be written, if yes, return to step 3040, if no, continue to step 3080;

[0067] Step 3080, merging multiple segment files in the same shard to generate shard data of the shard;

[0068] Since the search engine will merge all segment files under the same shard into one segment file, if they are not merged in advance, the search engine 21 will consume time to merge the segment files when loading data after opening the index.

[0069] To this end, in this embodiment, it is envisioned that all segment files under the same slice are merged into one segment file to save resources of the search engine 21 .

[0070] At this point, all the data to be written 20 are written to the distributed file system 22 to form shard data 1, shard data 2, ..., shard data n.

[0071] Step 3090, obtaining transaction log data corresponding to the shard data;

[0072] Since it is necessary to read the transaction log (translog) according to the shard data when opening the index, it is also necessary to obtain the transaction log corresponding to the shard data through step 3070 in this embodiment.

[0073] The transaction log includes two parameters: translog_generation and translog_uuid.

[0074] Currently, Elasticsearch does not have a corresponding API to obtain the contents of the above two parameters. Therefore, in this embodiment, a plug-in is implemented on Elasticsearch to obtain the contents of translog_generation and translog_uuid from internal information in Elasticsearch (for example, memory variables or log files).

[0075] Step 3100, storing the transaction log data in the shard corresponding to the shard data;

[0076] Specifically, the acquired transaction log is written into the first data shard.

[0077] Afterwards, in response to a request to write data, the following operations are performed:

[0078] Step 3110, controlling the search engine to close the index;

[0079] Step 3120, control the search engine to copy the shard data in the distributed file system to the folder corresponding to the shard corresponding to the index;

[0080] Copy the first shard data to the first folder; copy the second shard data to the second folder; ...; copy the N shard data to the Nth folder.

[0081] In this embodiment, the shard data is copied from the hadoop file system to Elasticsearch through hadoop fs-get.

[0082] Step 3130, control the search engine to open the index.

[0083] In the existing solution, during the connection with the search engine, the sub-data 11, sub-data 12 and sub-data 13 are written to the first folder corresponding to the first shard through the software interface, and three segment files are generated in the first folder corresponding to the first shard, and then they are merged into one segment file. Since the speed of creating data using the software interface is very slow, and multiple writing and merging operations need to be performed, the process of writing data is extremely cumbersome and time-consuming.

[0084] In the embodiment of the present application, when offline, the data writing tool of the search engine is used to create the shard data in the distributed file system 22 in advance, and the segment data corresponding to the shard data is merged; when connected to the search engine, only the file copy command is required, thereby greatly simplifying the process of writing data to the search engine and greatly shortening the time of writing data. The file copy command can write a large amount of data to the search engine at one time, and correspondingly overcomes the problem that the search engine does not support parallel writing of a large amount of data.

[0085] exist Figure 4 In another embodiment of the present application shown in FIG. 1 , in order to further improve the efficiency of data writing, Figure 3 The process shown has been improved.

[0086] refer to Figure 4 In the embodiment of the present application, Spark distributes the data to be written stored in Hadoop's file system HDFS1 to Spark's branch threads (partitions); each Spark's branch thread generates an independent Lucene instance, and the Lucene instance is responsible for writing data corresponding to a specific shard in the index to Hadoop's file system HDFS2; in addition, when copying the shard data from Hadoop's file system to the search engine, each shard data in Hadoop's file system HDFS2 can also be copied to the folder corresponding to the corresponding shard in the search engine in parallel through the corresponding Spark's branch threads.

[0087] The following process is executed in each branch thread in Spark:

[0088] Acquire the first sub-data in the data to be written;

[0089] Obtain the shard corresponding to the first sub-data to obtain the first shard;

[0090] Writing the first sub-data into the first shard data corresponding to the first shard through the Lucene instance to form a segment file of the first shard data;

[0091] Merge all segment files in the same shard data into one segment file;

[0092] Obtain the transaction log corresponding to the shard data to obtain the first transaction log;

[0093] The first transaction log is stored in the first shard data.

[0094] exist Figure 4 In the embodiment of the present application shown, by introducing Spark and parallel computing, the data writing method of the present application can be performed in parallel through multiple threads. In this way, the time required to write data to the search engine can be further greatly shortened, and the efficiency is higher.

[0095] Similarly, based on the above data writing method, the present application embodiment also provides a data writing device, such as Figure 5As shown, the device 50 includes: a closing index module 501, used to control the search engine to close the index corresponding to the data to be written, the index is used to index the shard data in multiple shards; a data copying module 502, used to control the search engine to copy the shard data corresponding to the shard of the index in the distributed file system to the folder corresponding to the corresponding shard of the index, the shard data is pre-written into the distributed file system according to the data to be written and the shard of the index by the data writing tool of the search engine; an opening index module 503, used to control the search engine to open the index.

[0096] According to one embodiment of the present application, the device 50 also includes: an index writing module, which is used to control the search engine to create an index corresponding to the data to be written; an index information acquisition module, which is used to obtain shard information corresponding to the index, and the shard information includes the number of shards, and the mapping relationship between the shard data and the number of shards.

[0097] According to one embodiment of the present application, the device 50 also includes: a data writing tool determination module, used to determine the data writing tool; a data offline writing module, used to write the data to be written to the distributed file system through the data writing tool to form shard data corresponding to the shard of the index.

[0098] According to an embodiment of the present application, the data offline writing module includes: a sub-data acquisition sub-module, used to determine multiple first sub-data in the data to be written and the shard information corresponding to each first sub-data; a shard determination sub-module, used to obtain the shard corresponding to the first sub-data to obtain the first shard; a sub-data writing sub-module, used to write each first sub-data into the corresponding shard through a data writing tool to form a first segment file of the first shard; a segment file merging module, used to merge multiple first segment files located in the same shard to generate shard data of the shard.

[0099] According to an embodiment of the present application, the device 50 further includes: a transaction log adding module, which is used to obtain transaction log data corresponding to the shard data, and store the transaction log data in the shard corresponding to the shard data.

[0100] According to an embodiment of the present application, the data offline writing module is specifically used to create threads with the same number as the number of shards, and the threads write the data to be written to the distributed file system one-to-one and concurrently through the data writing tool to form shard data corresponding to the shards of the index.

[0101] Similarly, based on the data writing method, an embodiment of the present application also provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements any of the above-mentioned data writing methods when executing the program.

[0102] Similarly, based on the data writing method, an embodiment of the present application also provides a computer-readable storage medium, which stores a program. When the program is executed by a processor, the processor executes at least any one of the above-mentioned data writing methods.

[0103] It should be pointed out here that the above description of the embodiment of the device for writing data to the search engine, the above description of the embodiment of the computer device, and the above description of the embodiment of the computer-readable storage medium are similar to the description of the aforementioned method embodiment, and have similar beneficial effects as the method embodiment, so they are not described in detail. For technical details not disclosed in the embodiment of the display device for configuring information of this application, please refer to the description of the aforementioned method embodiment of this application for understanding, and in order to save space, they are not described in detail.

[0104] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0105] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0106] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0107] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0108] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0109] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0110] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data writing method, characterized in that: The method comprises: in response to a request to write data, performing the following operations: Control the search engine to close the index corresponding to the data to be written, where the index is used to index the shard data in the multiple shards; Controlling the search engine to copy the shard data corresponding to the shard of the index in the distributed file system to the folder corresponding to the corresponding shard of the index, wherein the shard data is pre-written into the distributed file system by the data writing tool of the search engine according to the data to be written and the shard of the index; wherein the data writing tool of the search engine is used in an offline state to pre-write into the distributed file system according to the data to be written and the shard of the index; The search engine is controlled to open the index.

2. The method according to claim 1, characterized in that The method further comprises: Control the search engine to create the index corresponding to the data to be written; The shard information corresponding to the index is obtained, wherein the shard information includes the number of shards and a mapping relationship between the shard data and the number of shards.

3. The method according to claim 2, characterized in that The method further comprises: Determining the data writing tool; The data to be written is written into the distributed file system through the data writing tool to form shard data corresponding to the shards of the index.

4. The method according to claim 3, characterized in that The step of writing the data to be written into the distributed file system through the data writing tool to form shard data corresponding to the shards of the index includes: Determine a plurality of first sub-data in the data to be written and shard information corresponding to each first sub-data; Using the data writing tool, write each of the first sub-data into a corresponding shard to form a first segment file of the first shard; Merge multiple first segment files in the same shard to generate shard data for the shard.

5. The method according to claim 3, characterized in that: The method further comprises: Acquire transaction log data corresponding to the shard data, and store the transaction log data in the shard corresponding to the shard data.

6. The method according to claim 3, characterized in that The step of writing the data to be written into the distributed file system through the data writing tool to form shard data corresponding to the shards of the index includes: Create threads with the same number as the number of shards, and use the threads to concurrently write the data to be written into the distributed file system one-to-one through the data writing tool to form shard data corresponding to the shards of the index.

7. A data writing device, characterized in that: The device comprises: A closing index module, used to control the search engine to close the index corresponding to the data to be written, wherein the index is used to index the shard data in multiple shards; A data replication module, used to control the search engine to copy the shard data corresponding to the shard of the index in the distributed file system to the folder corresponding to the corresponding shard of the index, wherein the shard data is pre-written into the distributed file system by the data writing tool of the search engine according to the data to be written and the shard of the index; wherein the data writing tool of the search engine is used in an offline state to pre-write the data to be written into the distributed file system according to the shard of the index; The index opening module is used to control the search engine to open the index.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images