A data index construction system, method, storage medium and electronic device

Abstract

The present specification discloses a data index construction system, method, storage medium and electronic device, the system includes a data storage subsystem, a data processing subsystem, and an index construction subsystem, the data processing subsystem includes a data monitoring module, a message queue and a processing thread. The data storage subsystem can store data. The data monitoring module can determine the data change information when monitoring the data changes stored in the data storage subsystem, and store the data change information in the message queue. When the processing thread monitors that the message queue stores data change information, it can obtain the data change information from the message queue, process the data change information, obtain the index construction information in the specified format, generate an index construction request corresponding to the index construction information, and send it to the index construction subsystem. The index construction subsystem builds the target index according to the received index construction request. By monitoring data changes in real time, the efficiency of index construction is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data index construction system, method, storage medium and electronic device. Background Art

[0002] With the continuous development of science and technology, the total amount of data stored in data storage systems is growing exponentially, and how to efficiently access this massive data has received widespread attention.

[0003] At present, the massive data stored in the storage system itself is generally accessed based on the search function provided by the storage system itself. For example, when the storage system is a file storage system, the massive data stored in the text storage system can be accessed by scanning. However, some storage systems themselves may not be able to provide a search function, or the search function provided by the storage system itself may be relatively simple, resulting in low search efficiency. Therefore, in order to quickly search for data, an independent index system can also be built, and then the massive data stored in the storage system can be accessed based on the index system. However, building an index system takes a long time and is costly. Therefore, how to quickly build a data index is a very important issue.

[0004] Based on this, this specification provides a data index construction system. Summary of the invention

[0005] This specification provides a data index construction system, method, storage medium and electronic device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a method for constructing a data index, wherein the system includes a data storage subsystem, a data processing subsystem and an index construction subsystem, wherein the data processing subsystem includes a data monitoring module, a message queue and a processing thread; wherein:

[0008] The data storage subsystem is used to store data; wherein the storage method of the data storage subsystem includes at least one of object storage and file storage;

[0009] The data monitoring module is used to determine data change information when monitoring data changes stored in the data storage subsystem, and store the data change information in the message queue;

[0010] The processing thread is used to obtain the data change information from the message queue when monitoring that the data change information is stored in the message queue, process the data change information, obtain index building information in a specified format, generate an index building request corresponding to the index building information, and send the index building request to the index building subsystem;

[0011] The index building subsystem is used to receive the index building request and build a target index according to the index building request.

[0012] Optionally, the data processing subsystem includes a plurality of processing threads, the message queue includes a plurality of partitions, and there is a one-to-one correspondence between the partitions and the processing threads;

[0013] The data monitoring module is specifically used to determine the partition corresponding to the data change information among the several partitions included in the message queue, and use it as the target partition, storing the data change information in the target partition, so that when the processing thread corresponding to the target partition monitors that the data change information is stored in the target partition, it obtains the data change information from the target partition.

[0014] Optionally, the processing thread includes a preprocessing submodule and a request generation submodule;

[0015] The preprocessing submodule is used to obtain the data change information from the message queue when monitoring that the data change information is stored in the message queue, process the data change information, and obtain index building information in a specified format;

[0016] The request generation submodule is used to generate an index building request corresponding to the index building information, and send the index building request to the index building subsystem.

[0017] Optionally, the preprocessing submodule is specifically used to parse the data change information to determine first information in a preset format; determine an index strategy corresponding to the first information, and process the first information according to the index strategy to obtain index construction information in a specified format.

[0018] Optionally, the index building subsystem includes several coordination nodes and several data nodes;

[0019] The request generation submodule is specifically used to generate an index building request corresponding to the index building information; determine the coordinating node corresponding to the index building request from each coordinating node included in the index building subsystem, and send the index building request to the determined coordinating node;

[0020] The coordination node is used to respond to the index building request, determine the data identifier included in the index building request, determine the data node corresponding to the index building request from each data node included in the index building subsystem using a preset routing algorithm according to the data identifier, and send the index building request to the determined data node;

[0021] The data node is used to build a target index according to the received index building request.

[0022] Optionally, the pre-processing submodule is further configured to generate a batch processing instruction when it is monitored that the data change information stored in the message queue reaches a preset first threshold, and send the batch processing instruction to the request generating submodule;

[0023] The request generation submodule is also used to respond to the batch processing instruction, execute a preset waiting instruction, and when the received index building information reaches a preset second threshold, generate an index building request based on all the received index building information and send it to the index building subsystem.

[0024] This specification provides a method for constructing a data index, which is applied to a processing thread of a data processing subsystem in a data index construction system, wherein the data index construction system includes a data storage subsystem, the data processing subsystem and an index construction subsystem, and the data processing subsystem includes a data monitoring module, a message queue and the processing thread; the method includes:

[0025] monitoring the message queue, and when monitoring that data change information is stored in the message queue, obtaining the data change information from the message queue; wherein the data change information is determined by the data monitoring module when monitoring data changes stored in the data storage subsystem, the data storage subsystem is used to store data, and the storage mode of the data storage subsystem includes at least one of object storage and file storage;

[0026] The data change information is processed to obtain index building information in a specified format, an index building request corresponding to the index building information is generated, and the index building request is sent to the index building subsystem so that the index building subsystem receives the index building request and builds a target index according to the index building request.

[0027] Optionally, the data change information is processed to obtain index building information in a specified format, specifically including:

[0028] Parsing the data change information to determine first information in a preset format;

[0029] An indexing strategy corresponding to the first information is determined, and the first information is processed according to the indexing strategy to obtain index building information in a specified format.

[0030] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for constructing the data index is implemented.

[0031] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data index construction method when executing the program.

[0032] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0033] The data index construction system provided in this specification has a data storage subsystem that can store data, and the storage method of the data storage subsystem includes at least one of object storage and file storage. The data monitoring module can determine the data change information when monitoring the data changes stored in the data storage subsystem, and store the data change information in the message queue. When monitoring that the message queue has data change information stored, the processing thread can obtain the data change information from the message queue, process the data change information, obtain the index construction information in a specified format, generate an index construction request corresponding to the index construction information, and send the index construction request to the index construction subsystem. The index construction subsystem can receive the index construction request and build the target index according to the index construction request.

[0034] It can be seen from the above-mentioned data index construction system that when building an index, the data storage subsystem of the present application can store data, and the storage method of the data storage subsystem includes at least one of object storage and file storage. The data monitoring module can determine the data change information when monitoring the data change stored in the data storage subsystem, and store the data change information in the message queue. When the processing thread monitors that the message queue stores data change information, it can obtain the data change information from the message queue, process the data change information, obtain the index construction information in the specified format, generate the index construction request corresponding to the index construction information, and send the index construction request to the index construction subsystem. The index construction subsystem can receive the index construction request and build the target index according to the index construction request. The data monitoring module monitors the changes of data in the data storage subsystem in real time, and when monitoring the data change, the data change information is stored in the message queue in real time, so that the processing thread can quickly process the data change information, and send the index construction request corresponding to the processed information to the index construction subsystem, so that the index construction subsystem can quickly build the index. In addition, the data monitoring module can monitor data stored in multiple ways, obtain data change information, uniformly process multi-dimensional data, and synchronize data change information to the processing thread, making it suitable for the current requirements and trends of unified storage management of massive data stored in multiple storage methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0036] Figure 1 A schematic diagram of a data index construction system provided in this specification;

[0037] Figure 2 A schematic diagram of a process for determining index building information provided in this specification;

[0038] Figure 3 A schematic diagram of building an index for an index building subsystem provided in this specification;

[0039] Figure 4 A schematic diagram of the structure of a processing thread provided in this specification;

[0040] Figure 5 A schematic diagram of a method for constructing a data index provided in this specification;

[0041] Figure 6 A flowchart of another method for constructing a data index provided in this specification;

[0042] Figure 7 A method corresponding to the Figure 5 or Figure 6 Schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0044] At present, with the continuous development of science and technology, social production and life are ushering in the era of big data and AI. The total amount of data stored in data storage systems is exponentially growing. Data storage systems usually use large-capacity disks and distributed architectures to support the storage of massive data. How to efficiently access and obtain these stored massive data has attracted widespread attention, and also puts forward high requirements for the supporting data retrieval capabilities.

[0045] Conventional data search is to access the massive data stored in the storage system itself based on the search function provided by the storage system itself. For example, when the storage system is a file storage system, the massive data stored in the text storage system can be accessed by scanning. However, the search function provided by the storage system itself may be relatively simple, resulting in relatively low search efficiency. For another example, the storage system may be an object storage system, but the storage system generally does not support the search function. In addition, the above method of accessing data based on the search function provided by the storage system itself is only suitable for simple scenarios with small data volumes, and it is difficult to meet the needs of large data volumes and multiple types of distributed storage systems.

[0046] In order to quickly search for data, you can also build an independent index system for all kinds of existing data, and then extract data from various data sources as needed, and build index data according to the required strategy to support search. This method is often bound to the application, and requires manual intervention by developers for customized development and configuration. In addition, batch processing is generally used for data extraction, which makes it difficult to ensure the real-time nature of index generation, and data scanning will cause additional pressure on the storage system load.

[0047] Based on this, the present specification provides a data index construction system, method, storage medium and electronic device. The technical solutions provided by each embodiment of the present specification are described in detail below in conjunction with the accompanying drawings.

[0048] Figure 1 is a schematic diagram of a data index construction system provided in this specification, Figure 1 The system shown includes a data storage subsystem 100 , a data processing subsystem 101 and an index building subsystem 102 . The data processing subsystem 101 includes a data monitoring module 1011 , a message queue 1012 , and a processing thread 1013 .

[0049] First, the data storage subsystem 100 can be used to store data. Among them, the storage method of the data storage subsystem 100 includes at least one of object storage and file storage, and the data storage subsystem 100 is a storage system for storing massive data. The data storage subsystem 100 can be a Ceph distributed storage system, which is an open source distributed storage system that provides object storage, block storage and file system storage functions. Of course, the data storage subsystem 100 can also be other existing distributed storage systems, and also provides two storage methods of object storage and file storage, which are not specifically limited in this specification. The data storage subsystem 100 can integrate and unify distributed storage resources, and provide file storage and object storage for collaborative access to multiple clients. The data storage subsystem 100 can also include a metadata node, a data storage node and a monitoring node, the metadata node is used to store metadata corresponding to the data, the data storage node is used to store data, and the monitoring node is used to monitor the status of the nodes in the data storage subsystem 100, which includes one of being used and unusable. The bottom layer of the data storage subsystem 100 organizes and manages basic data units through RADOS (Reliable AutonomicDistributed Object Store), and provides object storage and file storage upward through gateway services and file metadata services. The data storage subsystem 100 supports horizontal dynamic expansion of distributed resources and effectively meets the storage needs of massive data. The above data can be data stored in the data storage subsystem 100 through the client. The data can be any type of data, such as text data, image data, etc. The storage method of the data can be at least one of object storage and file storage. As for the specific storage method of the data, this specification does not make specific restrictions.

[0050] Then, the data monitoring module 1011 can be used to determine the data change information when monitoring the data stored in the data storage subsystem 100, and store the data change information in the message queue 1012. Among them, the above-mentioned data monitoring module 1011 can monitor whether the data stored in the data storage subsystem 100 has changed, and the change includes the addition, modification, deletion, etc. of data. The data change information may include a number of data fields and a number of field values. There is a one-to-one correspondence between the data fields and the field values, and the data fields included in the data change information corresponding to each storage method may be the same or different, and the data fields included in the data change information corresponding to each type of change may be the same or different. The above data change information at least includes the change type. The above-mentioned message queue 1012 is used to store data change information.

[0051] For the file storage storage mode, the change types corresponding to the data change information corresponding to the file storage include deleting files, creating files, modifying files, renaming and moving files. When the field value corresponding to the data field of the change type (i.e., op) is deleting files (i.e., delete), the data fields in the data change information also include the file system (i.e., file_system), the index node (i.e., inode), the path (i.e., path), and whether the data corresponding to the path is a file (i.e., is_file). When the field value corresponding to the data field of the change type is creating a file (i.e., create), the data fields in the data change information also include the file system, the index node, the name (i.e., name), the path, the data size (i.e., size), the creation time (i.e., create_time), the update time (i.e., update_time), and whether the data corresponding to the path is a file. When the field value corresponding to the data field of the change type is modifying files (i.e., modify), the data fields in the data change information also include the file system, the index node, the name, the path, the data size, the creation time, the update time, and whether the data corresponding to the path is a file. When the value of the data field corresponding to the change type is rename, the data field in the data change information also includes the file system, index node, name, path, data size, update time, and whether the data corresponding to the path is a file. When the value of the data field corresponding to the change type is move file, the data field in the data change information also includes the file system, index node, name, data size, update time, source path (src_path), destination path (dest_path), and whether the data corresponding to the path is a file.

[0052] For example, the data change information may be: {"file_system":"fileSystem1","path":" / dir1 / file1","op":"create","inode":"1099511728056","create_time":"2017-08-23 05:33:30.0.392642","size":"1024","update_time":"2017-08-23 05:33:30.0.392642s","name":"file1.txt","is_file":"1"}.

[0053] For the object storage storage method, the change types corresponding to the data change information stored in this file include creating objects and deleting objects. When the field value corresponding to the data field of the change type (i.e., op) is creating an object (i.e., create), the data fields in the data change information also include the object bucket (i.e., bucket), user (i.e., user), name (i.e., name), path (i.e., path), data size (i.e., size), and creation time (i.e., create_time). When the field value corresponding to the data field of the change type is deleting an object (i.e., delete), the data fields in the data change information also include the object bucket, user, name, and path.

[0054] In this specification, the data monitoring module 1011 can not only monitor the change information corresponding to the data stored in the file storage storage mode, but also monitor the change information corresponding to the data stored in the object storage storage mode. Based on this, when the data change stored in the data storage subsystem 100 is monitored, the data monitoring module 1011 can monitor the metadata server MDS log in the data storage subsystem 100, analyze the MDS log, determine the file operation information, and read the file operation content as the data change information. Among them, the MDS (MetadataServer) log is a log file used to record MDS operations and events in the file storage system (CephFS) in the Ceph distributed storage system. The content of the MDS log may include file operation information, error and warning information, performance indicators and other information. The file operation information includes information corresponding to operations such as file creation, deletion, and renaming. The information at least includes the data fields and field values ​​included in the data change information, which will not be repeated here. The error and warning information is an error or warning message generated when encountering an abnormal situation or potential problem. The performance indicators may include response time, throughput, etc., which are helpful in evaluating the performance of the MDS.

[0055] In addition, the data monitoring module 1011 can also monitor the RGW gateway in the data storage subsystem 100, analyze the RGW gateway log, determine the object operation information, and read the object operation content as data change information. Among them, the RGW (RADOS Gateway) gateway is an important component in the Ceph distributed storage system. The RGW (RADOS Gateway) gateway log is a log file generated by the RGW component in the Ceph distributed storage system, which is used to record the operation, events and performance information of the RGW. The RGW gateway log may include request records, response records, error and warning information, performance indicators and other information. The request record may include detailed information of the HTTP request, such as method (GET, PUT, DELETE, etc.), path, query parameters, HTTP header, etc. The response record is to record the response code, response time and other relevant information returned by the RGW gateway to the client. The error and warning information is an error or warning message generated when encountering an abnormal situation or potential problem. The performance indicator may include the number of requests, response time, throughput, etc., which helps to evaluate the performance of the RGW gateway.

[0056] In this specification, the message queue 1012 may also include several partitions, each data change information has a corresponding partition, that is, the data monitoring module 1011 may store the data change information in the partition corresponding to the data change information in the message queue 1012. Based on this, when the data change information is stored in the message queue, the data monitoring module 1011 may determine the partition corresponding to the data change information in the several partitions included in the message queue, and store the data change information in the target partition as the target partition, which helps to ensure the timing of the data change operation and index construction. Specifically, the data monitoring module 1011 may determine the data identifier according to the data change information. When the storage method is file storage, the data identifier may be an index node in the data change information, but when the storage method is object storage, the data identifier may also be a string corresponding to the path. Then, a pre-set first hash function is used to perform hash mapping on the data identifier to obtain a hash value corresponding to the data identifier. According to the hash value, the partition corresponding to the data change information is determined in the several partitions included in the message queue, and used as the target partition. The data change information is stored in the target partition. In addition, after determining the data identifier, the data monitoring module 1011 can determine the partition corresponding to the data change information among the partitions included in the message queue according to the pre-established mapping table, and use it as the target partition. The data change information is stored in the target partition. The mapping table is pre-established, and the mapping table includes the corresponding relationship between the data identifier and the partition.

[0057] Afterwards, the processing thread 1013 can be used to obtain the data change information from the message queue 1012 when it is monitored that the message queue 1012 stores data change information, process the data change information, obtain the index building information in the specified format, generate an index building request corresponding to the index building information, and send the index building request to the index building subsystem 102. Among them, the above-mentioned processing thread 1013 can monitor whether the above-mentioned message queue 1012 stores data change information, and if so, directly obtain the data change information from the message queue 1012. The specified format is a pre-set format, the index building information includes a number of fields and a number of field values, the format of the index building information is a specified format, and the format is related to the change type, that is, each change type has a corresponding format. In other words, the above-mentioned specified format includes several sub-formats corresponding to the change type. It should be noted that the specified format is a format supported by the following index building subsystem 102, that is, the following index building subsystem 102 can build an index based on the index building information in the specified format. The above-mentioned index building request carries the index building information.

[0058] The above-mentioned specified format includes several sub-formats corresponding to the change types. Specifically, taking the change type corresponding to the storage mode of file storage as an example, when the change type is to create a file, the sub-format corresponding to the creation of the file may be as follows:

[0059] POST / index1 / _doc / id

[0060] {"name":"value",

[0061] "path": "value",

[0062] "size": "value",

[0063] "create_time":"value",

[0064] "update_time":"value",

[0065] "content": "value"}

[0066] Among them, POST means to create an index, / index1 / _doc / id is the URL path, index1 means the index set corresponding to the above data identifier, and the index set is preset. _doc means the document type. id means the data identifier. The fields name, path, size, create_time, update_time, and content are the fields required to be included in the sub-format corresponding to the above-mentioned file creation, that is, the fields required to be included in the index construction information corresponding to the file creation. The above value is the field value corresponding to these fields, and the field value can be determined according to the above-mentioned data change information or the first information and the read content. The above content means the file content, and the value corresponding to the content is the read content. If necessary, the value corresponding to the content can be empty.

[0067] When the change type is to delete a file, the sub-format corresponding to the deletion file may be as follows:

[0068] DELETE / index1 / _doc / id

[0069] DELETE means deleting the index.

[0070] When the change type is to modify a file, the sub-format corresponding to the modified file may be as follows:

[0071] POST / index1 / _update / id

[0072] {"doc":{

[0073] "name": "value",

[0074] "path": "value",

[0075] "size": "value",

[0076] "create_time":"value",

[0077] "update_time":"value",

[0078] "content": "value"}

[0079] }

[0080] Among them, _update is an update or modification API provided by the index building subsystem 102 described below. doc is an object, and the fields and field values ​​(i.e., key-value pairs) in the doc object are the fields and field values ​​that need to be updated or modified, while the fields and field values ​​that are not in the doc object remain unchanged. It should be noted that the fields and field values ​​contained in the above doc object are all examples.

[0081] When the change type is rename, the sub-format corresponding to the rename can be as follows:

[0082] POST / index1 / _update / id

[0083] {"doc":{

[0084] "name": "value",

[0085] "path": "value",

[0086] "update_time":"value"}

[0087] }

[0088] It should be noted that the fields and their field values ​​contained in the above doc object are all examples.

[0089] When the change type is to move a file, the sub-format corresponding to the move file may be as follows:

[0090] POST / index1 / _update / id

[0091] {"doc":{

[0092] "name": "value",

[0093] "path": "dest_path",

[0094] "size": "value",

[0095] "update_time":"value"}

[0096] }

[0097] Among them, dest_path indicates the destination path.

[0098] When the data change information is processed as above and the index building information of the specified format is obtained, the processing thread 1013 can parse the data change information to determine the first information of the preset format. Then determine the index strategy corresponding to the first information, and process the first information according to the index strategy to obtain the index building information of the specified format. Among them, the preset format can be a preset format, and the preset format can be JSON format. The first information is information in JSON format. The index strategy can be a preset strategy, and the index strategy at least includes metadata index, and of course, can also include full-text index. When the index strategy includes full-text index, the processing method of the first information based on the index strategy is to read all the contents of the changed data when the changed data is a text file, and process the first information and the read content into the index building information of the specified format. Of course, the index strategy can also be empty, and this specification does not make specific restrictions. The changed data can be the data corresponding to the data change information monitored by the data monitoring module 1011.

[0099] Specifically, taking the above-mentioned index strategy including full-text indexing as an example, when the first information is processed according to the index strategy to obtain the index construction information of the specified format, the processing thread 1013 can first obtain the field value corresponding to the name data field from the first information, and then determine whether the suffix included in the name is a text file. If so, call the data reading interface to read the content of the file corresponding to the name, and then process the first information and the read content to obtain the index construction information of the specified format.

[0100] In this specification, the message queue 1012 includes several partitions. In order to quickly build an index, the system may also include several processing threads 1013, and each processing thread 1013 works in parallel. That is, there is a one-to-one correspondence between the partition and the processing thread 1013. Each processing thread 1013 monitors the partition corresponding to the message queue 1012 according to the above process, and when the partition has data change information, the data change information is processed to obtain the index building information in the specified format. Generate an index building request corresponding to the index building information, and send the index building request to the index building subsystem 102. Specifically, Figure 2 As shown, Figure 2 A schematic diagram of a process for determining index building information provided in this specification, Figure 2 The message queue 1012 in includes several partitions (i.e. Figure 2Partition 1 to partition n in the partition), also includes a number of processing threads 1013, each partition has a corresponding processing thread 1013, each processing thread 1013 monitors its corresponding partition, generates an index building request corresponding to the determined index building information, and sends the index building request to the index building subsystem 102. Through multiple processing threads 1013, they simultaneously wait for data changes in the monitoring message queue 1012. When new data arrives (that is, when the corresponding partition is monitored to have new data change information), the data change information is first parsed to determine the first information in a preset format. Then determine the index strategy corresponding to the first information, and according to the index strategy, process the first information to obtain the index building information in a specified format. The specific process is similar to the above-mentioned process of processing the data change information to obtain the index building information in a specified format, and will not be repeated here.

[0101] Specifically, the data monitoring module 1011 can first determine the partition corresponding to the data change information among the partitions included in the message queue 1012, and store the data change information in the target partition as the target partition. Afterwards, when the processing thread 1013 corresponding to the target partition monitors that the data change information is stored in the target partition, it obtains the data change information from the target partition. Subsequently, the data change information is processed to obtain the index building information in the specified format, generate an index building request corresponding to the index building information, and send the index building request to the index building subsystem 102.

[0102] Finally, the index building subsystem 102 can be used to receive an index building request and build a target index according to the index building request. Among them, the index building subsystem 102 can be an index system implemented based on Elasticsearch. The index building subsystem also provides query or search services. The Elasticsearch can optimize the coordination node settings, sharding strategies, etc. to meet the needs of efficient index synchronization of massive data. Elasticsearch is an open source distributed search and analysis engine suitable for various use cases, including full-text search, structured search, log analysis, security analysis, etc. Although the above-mentioned index building subsystem 102 is an index system implemented based on the existing Elasticsearch, the index building request input into the index building subsystem 102 is automatically generated by the above-mentioned processing thread 1013 according to the data change information, so that there is no need for the existing developers to manually write the index building request, saving personnel costs and time costs, thereby improving the efficiency of index building.

[0103] In this specification, the index construction subsystem 102 may include a number of coordination nodes 1021 and a number of data nodes 1022, such as Figure 3 As shown, Figure 3A schematic diagram of an index building subsystem provided in this specification building an index, Figure 3 It includes several coordination nodes 1021 (i.e. Figure 3 Coordinating nodes 1 to n) and several data nodes 1022 (i.e. Figure 3 Data nodes 1 to data nodes n in the table), it should be noted that: Figure 3 This is only an example of index building subsystem building an index. The above processing thread 1013 can generate an index building request corresponding to the index building information, and then determine the coordinating node 1021 corresponding to the index building request from the coordinating nodes 1021 included in the index building subsystem 102, and send the index building request to the determined coordinating node 1021 (i.e. Figure 3 The index building request carries index building information. When determining the coordinating node 1021 corresponding to the index building request from the coordinating nodes 1021 included in the index building subsystem 102, the processing thread 1013 may use a preset second hash function to perform hash mapping on the data identifier in the index building request to obtain a hash value corresponding to the data identifier, and according to the hash value, determine the coordinating node 1021 corresponding to the index building request from the coordinating nodes 1021 included in the index building subsystem 102. The coordinating node 1021 submitted to the index building subsystem 102 is determined by a hash method to ensure load balancing of index data construction and improve construction efficiency.

[0104] Afterwards, the coordination node 1021 may respond to the index building request, determine the data identifier included in the index building request, and determine the data node 1022 (i.e. Figure 3 1), and sends the index building request to the determined data node 1022. The routing algorithm may be preset, and the routing algorithm may be modulo the number of data nodes included in the index building subsystem 102 for the data identifier. The index building subsystem 102 sets multiple coordination nodes 1021 to evenly distribute the received index building requests to achieve a balance between the network request load and the internal data replication migration to improve the index building efficiency.

[0105] Finally, the data node 1022 can build the target index according to the received index building request. Specifically, the data node 1022 can build the target index according to the pre-set building strategy and the received index building request, using a pre-set index building algorithm. The building strategy is pre-set, and the building strategy includes setting the index field strategy, word segmentation rules, and sharding strategy. The index building algorithm can be a pre-set algorithm. The index building algorithm is used to build the index.

[0106] It can be seen from the above data index construction system that the data storage subsystem can store data, and the storage method of the data storage subsystem includes at least one of object storage and file storage. The data monitoring module can determine the data change information when monitoring the data changes stored in the data storage subsystem, and store the data change information in the message queue. When the processing thread monitors that the message queue stores data change information, it can obtain the data change information from the message queue, process the data change information, obtain the index construction information in the specified format, generate the index construction request corresponding to the index construction information, and send the index construction request to the index construction subsystem. The index construction subsystem can receive the index construction request and build the target index according to the index construction request. The data monitoring module monitors the changes of the data in the data storage subsystem in real time, and when the data changes are monitored, the data change information is stored in the message queue in real time, so that the processing thread can quickly process the data change information, and send the index construction request corresponding to the processed information to the index construction subsystem, so that the index construction subsystem can quickly build the index. In addition, the data monitoring module can monitor data stored in multiple ways, obtain data change information, uniformly process multi-dimensional data, and synchronize data change information to the processing thread, making it suitable for the current requirements and trends of unified storage management of massive data stored in multiple storage methods.

[0107] Furthermore, the above-mentioned data monitoring module can monitor the logs in the distributed storage system (i.e., the data storage subsystem) at the same time, can obtain data change information of file storage and object storage in real time, and support the unified construction of indexes for the full amount of stored data without affecting the IO performance of the original storage system. It is suitable for the requirements of unified storage management of massive data stored in various storage methods.

[0108] Furthermore, the message queue includes several partitions, and each partition has a corresponding processing thread, and each processing thread processes data change information in parallel. Through the index construction mechanism of concurrent processing threads and the collaborative working mode of multiple processing threads, the efficiency of data preprocessing and index construction is improved. And when determining the partition corresponding to the data change information, a hash mechanism can be used for mapping to determine the target partition, ensuring the timing of load balancing and data operation synchronization, and then ensuring the consistency of the index, greatly improving the efficiency of index data synchronization, which is suitable for large-scale user scenarios.

[0109] In this specification, the processing thread 1013 may also include a preprocessing submodule 10131 and a request generation submodule 10132, which may be specifically as follows: Figure 4 As shown, Figure 4 The schematic diagram of the structure of a processing thread provided in this specification is that the preprocessing submodule 10131 can be used to obtain the data change information from the message queue 1012 when the data change information is stored in the message queue 1012. The data change information is processed to obtain the index construction information in the specified format. Specifically, the preprocessing submodule 10131 can parse the data change information to determine the first information in the preset format. Then determine the index strategy corresponding to the first information, and process the first information according to the index strategy to obtain the index construction information in the specified format.

[0110] The request generation submodule 10132 may be used to generate an index building request corresponding to the index building information, and send the index building request to the index building subsystem 102 .

[0111] In this specification, the above-mentioned index building subsystem 102 includes a number of coordination nodes 1021 and a number of data nodes 1022. The above-mentioned request generation submodule 10132 can generate an index building request corresponding to the index building information. From the various coordination nodes 1021 included in the index building subsystem, the coordination node 1021 corresponding to the index building request is determined, and the index building request is sent to the determined coordination node 1021. Among them, the above-mentioned index building request carries index building information. When determining the coordination node 1021 corresponding to the index building request from the various coordination nodes 1021 included in the index building subsystem 102, the request generation submodule 10132 can use a pre-set second hash function to perform hash mapping on the data identifier in the index building request, and obtain a hash value corresponding to the data identifier. According to the hash value, the coordination node 1021 corresponding to the index building request is determined in the various coordination nodes 1021 included in the index building subsystem 102.

[0112] Afterwards, the coordination node 1021 is used to respond to the index building request, determine the data identifier included in the index building request, and based on the data identifier, use a pre-set routing algorithm to determine the data node 1022 corresponding to the index building request from among the data nodes 1022 included in the index building subsystem 102, and send the index building request to the determined data node 1022.

[0113] Finally, the data node 1022 may build a target index according to the received index building request.

[0114] In this specification, in order to avoid excessive load and reduce network overhead, the processing thread 1013 can send the index building request including multiple index building information to the index building subsystem 102 in a batch processing manner, so that the index building subsystem 102 can batch process the index building request including multiple index building information. Based on this, the processing thread 1013 can also execute the preset waiting instruction when the data change information stored in the monitored message queue 1012 reaches the preset first threshold, and when the determined index building information reaches the preset second threshold, generate the index building request corresponding to all the determined index building information, and then send the index building request to the index building subsystem 102. Among them, the first threshold and the second threshold are both preset thresholds, and the first threshold and the second threshold can be the same or different, which is not specifically limited in this specification. The waiting instruction is a preset instruction, and the waiting instruction is used to instruct the processing thread 1013 not to send the index building request including a single index building information generated in real time to the index building subsystem 102, but to send the index building request including all the index building information to the index building subsystem 102 when the determined index building information reaches the second threshold. The above index building request carries all the determined index building information.

[0115] Based on this, the pre-processing submodule 10131 can also generate a batch processing instruction when the data change information stored in the monitored message queue 1012 reaches a preset first threshold, and send the batch processing instruction to the request generation submodule 10132. Afterwards, the request generation submodule 10132 can also respond to the batch processing instruction, execute the preset waiting instruction, and when the received index building information reaches a preset second threshold, generate an index building request according to all the received index building information, and send it to the index building subsystem 102. Among them, the above-mentioned batch processing instruction is a preset instruction, and the batch processing instruction is used to instruct the request generation submodule 10132 to specify the preset waiting instruction.

[0116] In this specification, the index building subsystem 102 may also include a master node, which is responsible for managing all nodes and shards in the index building subsystem 102 and is also used to store building strategies. Therefore, the data node 1022 needs to first obtain the preset building strategy from the master node, and then generate the target index based on the building strategy.

[0117] In this specification, the above-mentioned data node 1022 may include at least one primary shard, which can be used to build a target index according to the received index building request. Based on this, the above-mentioned coordination node 1021 can also respond to the index building request, determine the data identifier included in the index building request, and according to the data identifier, use a pre-set routing algorithm to determine the primary shard corresponding to the index building request from each primary shard corresponding to each data node 1022 included in the index building subsystem 102, and use it as the target primary shard. Send the index building request to the data node 1022 where the target primary shard is located. Among them, the routing algorithm can be pre-set, and the routing algorithm can also take the modulus of the number of primary shards included in the index building subsystem 102 for the data identifier. Afterwards, after the data node 1022 where the target primary shard is located receives the index building request, the target primary shard can build the target index according to the index building request. Specifically, the target primary shard may first perform a word segmentation operation on the read content, and then establish an inverted index to construct a target index, and record the operations involved in the index construction process in a transaction log (Translog) so that it can be recovered from when a failure occurs.

[0118] In addition, the system will replicate data between the primary shard and the replica shard, and merge segments within the shard regularly. Therefore, after the primary shard has built the index, the index can also be copied to the replica shard to ensure data reliability and improve retrieval efficiency. The above-mentioned data node 1022 may also include replica shards, and each primary shard corresponds to at least one replica shard. Therefore, after the target primary shard has built the target index, the target index can be copied to at least one replica shard corresponding to the target primary shard, and at least one replica shard will return a confirmation message to the target primary shard after receiving the target index, so that the target primary shard will send a confirmation message to the coordination node after receiving the confirmation message. The confirmation message indicates that the backup of the target index is complete.

[0119] In this specification, the data storage subsystem 100 can be pre-built based on the Ceph distributed storage system. The index building subsystem can be pre-built based on the Elasticsearch distributed index system. The message queue 1012 can be pre-built based on Kafka, and the number of partitions in the message queue 1012 can be pre-set. The number of the processing threads 1013 is consistent with the number of partitions in the message queue 1012, and each processing thread 1013 monitors one partition.

[0120] In this specification, before building an index, storage resource synchronization can be performed. Specifically, the data storage subsystem 100 can create storage resources, which include storage resources of a file storage system and storage resources of an object storage system. When the data monitoring module 1011 monitors the change of the storage resources of the data storage subsystem 100, it obtains the metadata corresponding to the changed storage resources and stores the metadata in the message queue. When the processing thread 1013 monitors that the message queue stores metadata, it obtains the metadata from the message queue, processes the metadata, obtains the second information in a specified format, generates a setting request corresponding to the second information, and sends the setting request to the index building subsystem 102. The index building subsystem 102 receives the setting request information and sets the index set and the building strategy according to the setting request. Among them, the metadata corresponding to the above storage resources may include the identification, name, size, building strategy, index strategy and other information of the storage space, and the above index set and the storage space have a one-to-one correspondence. The above second information may be consistent with the content included in the above metadata. The above setting request carries the second information.

[0121] In this specification, the index building subsystem 102 also provides a search or query function, and users can use the search or query function provided by the index building subsystem 102 to search or query data based on the index built in the index building subsystem 102. The search or query function provided by the index building subsystem 102 supports retrieval by default index fields such as file name, directory, owner, etc., and of course, retrieval can also be based on the full-text index, which is obtained by extracting and building the full-text data of the text file, so that when facing massive data, the index building subsystem 102 provides a wealth of retrieval means and fast and efficient retrieval capabilities.

[0122] based on Figure 1 The data index construction system shown in the specification also provides a data index construction method, which is applied to the processing thread of the data processing subsystem in the data index construction system. The data index construction system includes a data storage subsystem, a data processing subsystem and an index construction subsystem. The data processing subsystem includes a data monitoring module, a message queue and a processing thread. Figure 5 As shown, Figure 5 A flowchart of a method for constructing a data index provided in this specification includes the following steps:

[0123] S100: monitor the message queue, and when it is monitored that data change information is stored in the message queue, obtain the data change information from the message queue; wherein the data change information is determined by the data monitoring module when it monitors the data change stored in the data storage subsystem, the data storage subsystem is used to store data, and the storage method of the data storage subsystem includes at least one of object storage and file storage.

[0124] S102: Process the data change information to obtain index building information in a specified format, generate an index building request corresponding to the index building information, and send the index building request to the index building subsystem, so that the index building subsystem receives the index building request and builds a target index according to the index building request.

[0125] The processing thread can monitor the message queue, and when it monitors that data change information is stored in the message queue, it obtains the data change information from the message queue, processes the data change information, obtains index building information in a specified format, generates an index building request corresponding to the index building information, and sends the index building request to the index building subsystem, so that the index building subsystem receives the index building request and builds the target index according to the index building request. The data change information is determined by the data monitoring module when it monitors the data change stored in the data storage subsystem, the data storage subsystem is used to store data, and the storage method of the data storage subsystem includes at least one of object storage and file storage.

[0126] The above-mentioned specified format is a pre-set format. The index building information includes several fields and several field values. The format of the index building information is a specified format, and the format is related to the change type, that is, each change type has a corresponding format. In other words, the above-mentioned specified format includes several sub-formats corresponding to the change type. The above-mentioned data change information is processed to obtain the index building information in the specified format and the above-mentioned Figure 1 The process of sending the index building request to the index building subsystem is similar to that in the data index building system shown in FIG. Figure 1 The process in the data index construction system shown is similar and will not be repeated here.

[0127] based on Figure 1 The data index construction system shown in the specification also provides a data index construction method, which is applied to the data monitoring module of the data processing subsystem in the data index construction system. The data index construction system includes a data storage subsystem, a data processing subsystem and an index construction subsystem. The data processing subsystem includes a data monitoring module, a message queue and a processing thread. Figure 6As shown, Figure 6 A flowchart of another method for constructing a data index provided in this specification includes the following steps:

[0128] S200: monitoring the data storage subsystem, and determining data change information when monitoring data changes stored in the data storage subsystem; wherein the data storage subsystem is used to store data, and the storage method of the data storage subsystem includes at least one of object storage and file storage.

[0129] S202: The data change information is stored in the message queue, so that when the processing thread monitors that the data change information is stored in the message queue, it obtains the data change information from the message queue, processes the data change information, obtains index building information in a specified format, generates an index building request corresponding to the index building information, and sends the index building request to the index building subsystem, so that the index building subsystem receives the index building request and builds the target index according to the index building request.

[0130] The data monitoring module can monitor the data storage subsystem, and when monitoring the data changes stored in the data storage subsystem, determine the data change information. Then store the data change information in the message queue, so that when the processing thread monitors the data change information stored in the message queue, it obtains the data change information from the message queue, processes the data change information, obtains the index building information in the specified format, generates an index building request corresponding to the index building information, and sends the index building request to the index building subsystem, so that the index building subsystem receives the index building request and builds the target index according to the index building request.

[0131] The data change information may include several data fields and several field values. There is a one-to-one correspondence between the data fields and the field values. The data fields included in the data change information corresponding to each storage mode may be the same or different. The data fields included in the data change information corresponding to each type of change may be the same or different. The above data change information at least includes the type of change. The specific process of determining the data change information when monitoring the data change stored in the data storage subsystem is the same as the above Figure 1 The process of building the data index in the system shown in FIG. 1 is similar to that in FIG. 1 , and will not be described in detail here. The specific process of storing the data change information in the message queue is also similar to that in the above Figure 1 The process in the data index construction system shown is similar and will not be repeated here.

[0132] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 5 or Figure 6 Provides a method for constructing a data index.

[0133] This manual also provides Figure 7 The one shown corresponds to Figure 5 or Figure 6 A schematic diagram of the electronic device. Figure 7 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 5 or Figure 6 A method for constructing a data index is provided.

[0134] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0135] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0136] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0137] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0138] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0139] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0144] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0145] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0146] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0147] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0149] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0150] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A data index construction system, characterized in that: The system includes a data storage subsystem, a data processing subsystem and an index building subsystem, wherein the data processing subsystem includes a data monitoring module, a message queue and a processing thread; wherein: The data storage subsystem is used to store data; wherein the storage method of the data storage subsystem includes at least one of object storage and file storage; The data monitoring module is used to determine data change information when monitoring data changes stored in the data storage subsystem, and store the data change information in the message queue; wherein the data change information at least includes a change type, and the change types contained in the change information corresponding to the object storage and the file storage are different, and the change type is used to indicate the type of data operation that causes the data change; The processing thread is used to obtain the data change information from the message queue when monitoring that the data change information is stored in the message queue, process the data change information, obtain index building information in a specified format, generate an index building request corresponding to the index building information, and send the index building request to the index building subsystem; The index building subsystem is used to receive the index building request and build a target index according to the index building request.

2. The system according to claim 1, characterized in that The data processing subsystem includes a plurality of processing threads, the message queue includes a plurality of partitions, and there is a one-to-one correspondence between the partitions and the processing threads; The data monitoring module is specifically used to determine the partition corresponding to the data change information among the several partitions included in the message queue, and use it as the target partition, storing the data change information in the target partition, so that when the processing thread corresponding to the target partition monitors that the data change information is stored in the target partition, it obtains the data change information from the target partition.

3. The system according to claim 1, characterized in that The processing thread includes a preprocessing submodule and an index building submodule; The preprocessing submodule is used to obtain the data change information from the message queue when monitoring that the data change information is stored in the message queue, process the data change information, and obtain index building information in a specified format; The request generation submodule is used to generate an index building request corresponding to the index building information, and send the index building request to the index building subsystem.

4. The system according to claim 3, characterized in that The preprocessing submodule is specifically used to parse the data change information to determine the first information in a preset format; determine the index strategy corresponding to the first information, and process the first information according to the index strategy to obtain index construction information in a specified format.

5. The system according to claim 3, characterized in that The index building subsystem includes several coordination nodes and several data nodes; The request generation submodule is specifically used to generate an index building request corresponding to the index building information; determine the coordinating node corresponding to the index building request from each coordinating node included in the index building subsystem, and send the index building request to the determined coordinating node; The coordination node is used to respond to the index building request, determine the data identifier included in the index building request, determine the data node corresponding to the index building request from each data node included in the index building subsystem using a preset routing algorithm according to the data identifier, and send the index building request to the determined data node; The data node is used to build a target index according to the received index building request.

6. The system according to claim 3, characterized in that The pre-processing submodule is further used to generate a batch processing instruction when it is monitored that the data change information stored in the message queue reaches a preset first threshold, and send the batch processing instruction to the request generation submodule; The request generation submodule is also used to respond to the batch processing instruction, execute a preset waiting instruction, and when the received index building information reaches a preset second threshold, generate an index building request based on all the received index building information and send it to the index building subsystem.

7. A method for constructing a data index, characterized in that: The method is applied to a processing thread of a data processing subsystem in a data index construction system, wherein the data index construction system includes a data storage subsystem, the data processing subsystem and an index construction subsystem, and the data processing subsystem includes a data monitoring module, a message queue and the processing thread; the method includes: Monitor the message queue, and when it is monitored that data change information is stored in the message queue, obtain the data change information from the message queue; wherein the data change information is determined by the data monitoring module when it is monitored that the data storage subsystem stores data changes, the data storage subsystem is used to store data, and the storage method of the data storage subsystem includes at least one of object storage and file storage; the data change information at least includes a change type, the change types contained in the change information corresponding to the object storage and the file storage are different, and the change type is used to indicate the type of data operation that causes the data to change; process the data change information to obtain index building information in a specified format, generate an index building request corresponding to the index building information, and send the index building request to the index building subsystem, so that the index building subsystem receives the index building request and builds a target index according to the index building request.

8. The method according to claim 7, characterized in that The data change information is processed to obtain index building information in a specified format, specifically including: Parsing the data change information to determine first information in a preset format; An indexing strategy corresponding to the first information is determined, and the first information is processed according to the indexing strategy to obtain index building information in a specified format.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 7 to 8 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 7 to 8 is implemented.

Images