Index retrieval method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an index retrieval method and device, electronic equipment and a computer readable storage medium, index data construction is performed on original data through an index construction module to obtain index structured data, and then the index structured data is transferred into a target memory to obtain index structured data. The retrieval module is prevented from directly obtaining the index structured data constructed by the index construction module, and the construction module and the retrieval module are retrieved in an off-line mode. And finally, performing index retrieval processing on the index structured data by adopting a retrieval module which is mutually independent from the index construction module, and realizing decoupling of an index construction process and a retrieval process by the index construction module and the retrieval module which are mutually independent and unloading of the index structured data. The technical problem that index construction and retrieval in a search engine are coupled in the same module, so that resource competition occurs during index construction and retrieval operation, and the retrieval waiting time of the search engine is long is solved.

Description

Technical Field

[0001] The present application relates to the field of data indexing technology, and in particular to an index retrieval method, device, electronic device and computer-readable storage medium. Background Art

[0002] In recent years, with the rapid development of information technology, people can easily use search engines to obtain the information they need. In search engines, since index construction and retrieval are coupled in the same module, there will be resource competition during index construction and retrieval operation, resulting in a long search waiting time for search engines. Summary of the invention

[0003] The embodiments of the present application provide an index retrieval method, device, electronic device and computer-readable storage medium, which can solve the technical problem that index construction and retrieval in a search engine are coupled in the same module, resulting in resource competition during index construction and retrieval operations, leading to a long retrieval waiting time of the search engine.

[0004] In a first aspect, an embodiment of the present application provides an index retrieval method, the method comprising:

[0005] Transferring the index structured data from the index building module to the target memory; wherein the index structured data is obtained by the index building module building the index data on the original data;

[0006] In a data retrieval scenario, the index structured data in the target memory is used for index retrieval processing based on the retrieval module; wherein the index construction module and the retrieval module are independent of each other.

[0007] Optionally, transferring the index structured data from the index building module to the target memory includes:

[0008] Calling the index building module to push the index structured data in the index building module to the first message queue;

[0009] The index structured data is obtained from the first message queue, and the index structured data is stored in the target memory.

[0010] Optionally, transferring the index structured data from the index building module to the target memory includes:

[0011] Controlling the index transfer thread based on the index building module to transfer the index structured data in the index building module to the target memory;

[0012] In the data retrieval scenario, the index retrieval process is performed based on the retrieval module using the index structured data in the target memory, including:

[0013] In a data retrieval scenario, based on a retrieval thread corresponding to a retrieval module, index structured data in the target memory is used to perform index retrieval processing; wherein the index transfer thread and the retrieval thread share the target memory.

[0014] Optionally, the method further comprises:

[0015] Performing shared memory allocation processing on the index transfer thread and the retrieval thread to obtain a target memory;

[0016] The target memory is respectively subjected to content association processing with the index transfer thread and the retrieval thread.

[0017] Optionally, the acquiring the index structured data from the first message queue and storing the index structured data in a target memory includes:

[0018] Detect the memory occupancy ratio of the target memory. When the memory occupancy ratio is greater than or equal to a preset memory occupancy ratio threshold, transfer the index structured data in the target memory to a target disk associated with the target memory, and clean up the transferred index structured data in the target memory.

[0019] Optionally, the index structured data is obtained by the index building module performing index data construction on the original data, including:

[0020] Based on the index building module, the original data is preprocessed to obtain a forward index and a reverse index, and the forward index and the reverse index are structured to obtain index structured data.

[0021] Optionally, the index structured data is obtained by the index building module performing index data construction on the original data, including:

[0022] Based on at least two reference containers in the container cluster, the index building module is used to obtain raw data;

[0023] Based on each reference container, the index construction module is used to construct an index structure for the original data to obtain distributed index structured data corresponding to each reference container, and the distributed index structured data are merged to obtain index structured data.

[0024] Optionally, before acquiring the original data by using the index building module based on at least two reference containers in the container cluster, the method further includes:

[0025] At least two reference containers in the container cluster are determined, and the index construction module is deployed to the at least two reference containers.

[0026] Optionally, determining at least two reference containers in the container cluster includes:

[0027] Acquire the index data volume of the original data, and determine the reference container deployment information from the container cluster based on the index data volume;

[0028] At least two reference containers are determined from the container cluster based on the reference container deployment information.

[0029] Optionally, the acquiring original data by using the index building module based on at least two reference containers in the container cluster includes:

[0030] Acquire original data, and write the original data into a second message queue;

[0031] The corresponding index building modules in at least two reference containers in the container cluster are called to obtain the original data from the second message queue.

[0032] In a second aspect, an embodiment of the present application provides an index retrieval device, the device comprising:

[0033] An index building module, adapted to transfer the index structured data from the index building module to the target memory; wherein the index structured data is obtained by the index building module performing index data building on the original data;

[0034] The retrieval module is suitable for performing index retrieval processing based on the index structured data in the target memory in a data retrieval scenario; wherein the index construction module and the retrieval module are independent of each other.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0036] Processor; and

[0037] A memory arranged to store computer executable instructions which, when executed, cause the processor to perform a method according to any one of the above.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, implement any of the methods described above.

[0039] The beneficial effects brought by the technical solutions provided in some embodiments of the present application include at least: constructing index data of the original data through the index construction module to obtain index structured data, and then transferring the index structured data to the target memory to avoid the retrieval module directly obtaining the index structured data constructed by the index construction module, and using the offline index construction module and retrieval module. Finally, a retrieval module that is independent of the index construction module is used to perform index retrieval processing on the index structured data. The independent index construction module and retrieval module and the transfer of index structured data realize the decoupling of the index construction process and the retrieval process, and solve the technical problem that the index construction and retrieval in the search engine are coupled in the same module, so that there will be resource competition when the index construction and retrieval are running, resulting in a long retrieval waiting time for the search engine. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 An exemplary system architecture diagram of an index retrieval method provided in an embodiment of the present application;

[0042] Figure 2 A flowchart of an index retrieval method provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of a process for obtaining index structured data provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of a process for determining at least two reference containers provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a process for obtaining raw data provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of a process for transferring index structured data provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of a process for associating a target memory with an index transfer thread and a retrieval thread respectively provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of the architecture of an index building module provided in an embodiment of the present application;

[0049] Fig. 9 A schematic diagram of the architecture of an index retrieval method provided in an embodiment of the present application corresponding to an embodiment;

[0050] Fig.10 A schematic diagram of the structure of an index retrieval device provided in an embodiment of the present application;

[0051] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the embodiments of the present application.

[0053] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0054] With the rapid development of information technology, users can easily use search engines to query the information they need. In search engines, index construction and retrieval are coupled in the same module, which leads to resource competition during index construction and retrieval operation, resulting in a long search waiting time for search engines. In addition, since index construction and retrieval are coupled in the same module, the index library can only be built online, and offline construction cannot be achieved.

[0055] Here, database construction includes database construction in a narrow sense and database construction in a broad sense. Database construction in a narrow sense can be the process of generating terms (index words) from web pages to web page indexes. Database construction in a broad sense can be the generation of index files and positive ranking, summaries, snapshots and structured data, as well as auxiliary data preprocessing, data mining and other processing operations in the generation process. Positive ranking refers to the forward information from URL (Uniform Resource Locator) to term list, term weight and attributes, and term offset (offset) generated after the web page is parsed, segmented, and relevance calculated. The purpose of positive ranking can include: generating reverse ranking and making rank reference during retrieval. Among them, reverse ranking refers to the reverse generation of zipper lists from term to URL and reverse information such as term weight, attribute and offset in URL by flipping the positive ranking data.

[0056] In order to overcome the above-mentioned technical problems, an embodiment of the present application provides an index retrieval method, which first transfers the index structured data from the index construction module to the target memory; wherein the index structured data is obtained by the index construction module performing index data construction on the original data; in a data retrieval scenario, index retrieval processing is performed based on the index structured data in the target memory based on the retrieval module; wherein the index construction module and the retrieval module are independent of each other.

[0057] In the embodiment of the present application, the original data is indexed by the index construction module to obtain index structured data, and then the index structured data is transferred to the target memory to avoid the retrieval module directly obtaining the index structured data constructed by the index construction module, and the index construction module and the retrieval module are offline. Finally, a retrieval module that is independent of the index construction module is used to perform index retrieval processing on the index structured data. The independent index construction module and retrieval module and the transfer of index structured data realize the decoupling of the index construction process and the retrieval process, and solve the technical problem that the index construction and retrieval in the search engine are coupled in the same module, so that there will be resource competition when the index construction and retrieval are running, resulting in a long waiting time for the search engine.

[0058] See also Figure 1 , Figure 1 An exemplary system architecture diagram of an index retrieval method provided in an embodiment of the present application.

[0059] like Figure 1As shown, the system architecture may include an electronic device 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the electronic device 101 and the server 103. The network 102 may include various types of wired communication links or wireless communication links, for example, the wired communication link includes an optical fiber, a twisted pair, or a coaxial cable, and the wireless communication link includes a Bluetooth communication link, a Wireless-Fidelity (Wi-Fi) communication link, or a microwave communication link.

[0060] The electronic device 101 can interact with the server 103 through the network 102 to receive messages from the server 103 or send messages to the server 103, or the electronic device 101 can interact with the server 103 through the network 102 to receive messages or data sent by other users to the server 103. The electronic device 101 can be hardware or software. When the electronic device 101 is hardware, it can be various electronic devices, including but not limited to smart watches, smart phones, tablet computers, laptop portable computers and desktop computers. When the electronic device 101 is software, it can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules (for example: used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited here.

[0061] The server 103 may be a business server that provides various services. It should be noted that the server 103 may be hardware or software. When the server 103 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or it may be implemented as a single server. When the server 103 is software, it may be implemented as multiple software or software modules (for example, for providing distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0062] In an embodiment of the present application, the server 103 can transfer the index structured data from the index construction module to the target memory; wherein, the index structured data is obtained by the index construction module constructing the original data; in the data retrieval scenario, the index retrieval processing is performed based on the index structured data in the target memory based on the retrieval module; wherein, the index construction module and the retrieval module are independent of each other.

[0063] It should be understood that Figure 1 The number of electronic devices, networks and servers in the figure is only illustrative, and any number of electronic devices, networks and servers may be used according to implementation requirements.

[0064] See also Figure 2 , Figure 2A flowchart of an index retrieval method provided for an embodiment of the present application. The execution subject of the embodiment of the present application can be a server that executes the index retrieval method, or a processor in the server that executes the index retrieval method, or an index processing service in the server that executes the index retrieval method. For the convenience of description, the specific execution process of the index retrieval method is introduced below by taking the execution subject being a processor in the server as an example.

[0065] like Figure 2 As shown, the index retrieval method may at least include:

[0066] S202: Transferring the index structured data from the index building module to the target memory; wherein the index structured data is obtained by the index building module building index data on the original data.

[0067] First, the original data is obtained. The original data may include but is not limited to various web resources, such as social media web resources, e-commerce web resources, news web resources, and academic web resources. Web resources may be data in various formats such as text, pictures, and videos on a web page. In a feasible implementation, the original data may be obtained based on a web crawler; wherein a web crawler is a program or script that automatically crawls World Wide Web information according to certain rules.

[0068] After obtaining the original data, the original data is indexed based on the index construction module. The index construction includes but is not limited to preprocessing such as parsing, analyzing and word segmentation of the original data to establish a forward index and an inverted index, and converting the built forward index and inverted index into index structured data of the Protobuf structure. The Protobuf structure consists of three parts: message definition, message encoding and message decoding. The message definition uses a .Proto file to define the data structure. The file defines the message type (equivalent to the data structure) and the field (equivalent to the member variable). The message type can define a nested message structure to implement a complex data model; the message encoding serializes the data into a binary format according to the message definition. Protobuf provides implementations in multiple programming languages, such as C++, Java, Python, etc. The implementations of different programming language versions can serialize the data into the corresponding binary format, and different programming language versions can interoperate; the message decoding is used to parse the binary data into the corresponding data structure according to the message definition. Of course, index structured data can also include hash tables, ordered arrays, and binary trees.

[0069] After obtaining the index structured data from the index building module, the index structured data constructed is transferred to a preset target memory through the index building module.

[0070] S204: In a data retrieval scenario, index retrieval processing is performed based on the retrieval module using the index structured data in the target memory; wherein the index construction module and the retrieval module are independent of each other.

[0071] When the server receives a data retrieval request, the server calls the retrieval module to perform index retrieval processing based on the index structured data in the target memory.

[0072] In addition, the retrieval module can be located in the target memory. At this time, since the retrieval module and the index structured data share the target memory, the index structured data just transferred to the target memory can be retrieved and queried by the retrieval module in the target memory. The index construction module and the retrieval module are independent of each other, that is, the index construction module and the retrieval module will not interfere with each other, and there is no resource competition relationship between the two. In other words, the index construction module and the retrieval module can be deployed in different servers respectively.

[0073] In the embodiment of the present application, the original data is first indexed by the index construction module to obtain index structured data, and then the index structured data is transferred to the target memory to avoid the retrieval module directly obtaining the index structured data constructed by the index construction module, and the index construction module and the retrieval module are offline. Finally, a retrieval module that is independent of the index construction module is used to perform index retrieval processing on the index structured data. The independent index construction module and retrieval module and the transfer of index structured data realize the decoupling of the index construction process and the retrieval process, and solve the technical problem that the index construction and retrieval in the search engine are coupled in the same module, so that there will be resource competition when the index construction and retrieval are running, resulting in a long waiting time for the search engine.

[0074] In an embodiment provided in the present application, transferring the index structured data from the index building module to the target memory in S202 includes:

[0075] Controlling the index transfer thread based on the index building module to transfer the index structured data in the index building module to the target memory;

[0076] In S204, in the data retrieval scenario, the retrieval module in the target memory uses the index structured data in the target memory to perform index retrieval processing, including:

[0077] In the data retrieval scenario, based on the retrieval thread corresponding to the retrieval module, the index structured data in the target memory is used to perform index retrieval processing; wherein the index transfer thread and the retrieval thread share the target memory.

[0078] After obtaining the index structured data, the index structured data constructed is processed by the index construction module based on the index transfer thread, and then transferred to the target memory. Specifically, after determining that the construction of the index structured data is completed, the index construction module can send an index data transfer instruction to the index transfer thread. After receiving the index data transfer instruction, the index transfer thread transfers the index structured data constructed by the index construction module, so that the index construction module can be separated during subsequent retrieval. After the index structured data is transferred, the index structured data can be obtained from the target memory after the index data is transferred instead of being obtained from the index construction module.

[0079] When the server receives a data retrieval request, the server determines the retrieval thread that shares memory with the index dump thread through the retrieval module, and the retrieval thread that shares memory with the index dump thread can obtain the storage location of the index structured data obtained after the index dump thread dumps the index structured data. Therefore, during the index retrieval process, the retrieval thread can perform retrieval based on the index structured data.

[0080] In an embodiment of the present application, first, the original data is indexed by the index construction module to obtain index structured data. After that, the retrieval thread corresponding to the retrieval module uses the index structured data in the target memory to perform index retrieval processing, so as to avoid the retrieval module directly obtaining the index structured data constructed by the index construction module. The retrieval thread that shares the memory with the index transfer thread can quickly obtain the transferred index structured data with the offline index construction module and the retrieval module, thereby effectively improving the retrieval efficiency, so that the newly written index structured data can be retrieved and queried in real time, and realizes the decoupling of the index construction module and the retrieval module, avoiding the phenomenon of resource competition when the index construction and retrieval are in the same module, and solving the technical problem that the index construction and retrieval in the search engine are coupled in the same module, resulting in resource competition when the index construction and retrieval are running, resulting in a long retrieval waiting time of the search engine.

[0081] See also Figure 3 , Figure 3 A schematic diagram of a process for obtaining index structured data provided in an embodiment of the present application. In an embodiment provided in the present application, the index structured data in S202 is obtained by indexing the original data by the index construction module, including:

[0082] S302: Based on at least two reference containers in the container cluster, original data is acquired using an index building module.

[0083] The container cluster includes two or more reference containers. The reference container is a portable container that can receive applications and their dependent packages. The portable container can be published to a machine with a Linux or Windows operating system. The containers use a sandbox mechanism and there is no interface between them. Specifically, the reference container can be Docker, which is an open source application container engine, and the container cluster can be a Docker cluster.

[0084] At least two reference containers in the container cluster may be deployed with index construction modules. After the original data is obtained, the original data may be distributed to at least two reference containers in the container cluster. Different reference containers receive different original data, that is, the index construction module corresponding to each reference container consumes a portion of the original data to achieve complete consumption of the entire original data based on the reference container in the container cluster.

[0085] Optionally, before acquiring original data using the index building module based on at least two reference containers in the container cluster in S302, the method further includes:

[0086] At least two reference containers in the container cluster are determined, and the index building module is deployed to the at least two reference containers.

[0087] At least two reference containers in the container cluster can be distributedly deployed in different servers to avoid the situation where index construction and retrieval are both in one module, resulting in the index construction being unable to be distributed. After determining at least two reference containers in the container cluster, the index construction module can be deployed to at least two reference containers respectively to achieve distributed deployment of the index construction module, and the index construction modules deployed in different reference containers can be the same.

[0088] See also Figure 4 , Figure 4 A schematic diagram of a process for determining at least two reference containers provided in an embodiment of the present application. In an embodiment provided in the present application, determining at least two reference containers in a container cluster includes:

[0089] S402: Obtain the index data volume of the original data, and determine the reference container deployment information from the container cluster based on the index data volume.

[0090] Obtain the index data volume of the original data that needs to be indexed, so as to determine the appropriate reference container deployment information from the container cluster when the index data volume is large, moderate or small. The reference container deployment information can be understood as the reference container information of the target number of reference containers determined in the container cluster, and each reference container is deployed with an index construction module; wherein the target number is generally greater than or equal to 2.

[0091] In a feasible implementation, when the amount of index data is large, 20 reference containers in the container cluster can be determined, and the index construction modules can be deployed to the 20 reference containers respectively; when the amount of index data is moderate, 10 reference containers in the container cluster can be determined, and the index construction modules can be deployed to the 10 reference containers respectively; when the amount of index data is small, 2 reference containers in the container cluster can be determined, and the index construction modules can be deployed to the 2 reference containers respectively. Of course, in other embodiments, when the amount of index data is small, 1 reference container in the container cluster can be determined, and the index construction modules can be deployed to the 1 reference container respectively.

[0092] S404: Determine at least two reference containers from the container cluster based on the reference container deployment information.

[0093] At least two reference containers are determined from the container cluster based on the obtained reference container information of the target number of reference containers, and then the index construction module is deployed to the at least two reference containers.

[0094] The embodiment of the present application determines the number of index construction modules to be distributedly deployed in the reference container based on the index data volume of the original data, thereby realizing dynamic expansion of the index construction modules during the index data construction process. When the index data volume is large, the number of index construction modules can be increased to improve the index construction speed. When the index data volume is small, the number of index construction modules can be reduced to save energy consumption.

[0095] See also Figure 5 , Figure 5 A schematic diagram of a process for obtaining raw data provided in an embodiment of the present application. In an embodiment provided in the present application, in S302, raw data is obtained using an index building module based on at least two reference containers in a container cluster, including:

[0096] S502: Acquire original data, and write the original data into the second message queue.

[0097] The web crawler crawls data from the Internet in real time, and then pushes the crawled data to a screening module for data screening. The screening module screens the crawled data to filter out redundant information such as advertisements and navigation bars. The web crawler crawls data from the Internet in real time and obtains the original data after screening. After obtaining the original data, the original data is written into the second message queue. The second message queue can be a container for storing messages during the transmission of the message. Specifically, the second message queue can be a distributed message queue. The distributed message queue has an asynchronous communication mechanism for programs or components. Based on the distributed message queue, the sender and the receiver do not have to be online at the same time, and the sender and the receiver do not have to know each other.

[0098] S504: Call corresponding index building modules in at least two reference containers in the container cluster to obtain original data from the second message queue.

[0099] The corresponding index building module in each reference container respectively obtains different parts of the original data from the second message queue, so that the corresponding index building modules in at least two reference containers in the container cluster can obtain complete original data from the second message queue. In addition, when the network is interrupted, the original data can still be obtained from the second message queue.

[0100] S304: constructing an index structure for the original data using an index construction module based on each reference container to obtain distributed index structured data corresponding to each reference container, and merging the distributed index structured data to obtain index structured data.

[0101] After obtaining a portion of the original data, the index construction module in each reference container constructs an index structure for the obtained original data to obtain distributed index structured data corresponding to the obtained original data, and then merges the distributed index structured data corresponding to each reference container to obtain complete index structured data corresponding to the complete original data.

[0102] See also Figure 6 , Figure 6 A schematic diagram of a process for transferring index structured data provided in an embodiment of the present application. In an embodiment provided in the present application, transferring the index structured data from the index building module to the target memory in S202 includes:

[0103] S602: Calling the index building module to push the index structured data in the index building module to the first message queue.

[0104] After the index structured data is generated, the server calls the index construction module to push the index structured data in the index construction module to the first message queue. For example, the server calls the index construction modules corresponding to at least two reference containers to push the index structured data to the first message queue. The relevant description of the first message queue can refer to the second message queue, which will not be repeated here. It should be noted that the storage space of the first message queue and the second message queue are independent of each other.

[0105] S604: Obtain index structured data from the first message queue, and store the index structured data in the target memory.

[0106] The first message queue is used as a transfer container for index structured data. The index structured data is obtained from the first message queue. After the index structured data is obtained, the index structured data is stored in the target memory.

[0107] Specifically, after the index construction module corresponding to the reference container pushes the index structured data to the first message queue, the index construction module corresponding to the reference container can instruct the index dump thread to obtain the index structured data pushed by the corresponding index construction module from the first message queue, so that the index dump thread dumps the index structured data to the target memory. The target memory is a shared memory of the index dump thread and the retrieval thread, that is, the index dump thread and the retrieval thread can both access the target memory, so that the written index structured data can be queried by the retrieval thread in real time.

[0108] See also Figure 7 , Figure 7 A flowchart of associating a target memory with an index transfer thread and a retrieval thread respectively is provided in an embodiment of the present application.

[0109] like Figure 7 As shown, the method includes:

[0110] S702: Perform shared memory allocation processing on the index transfer thread and the retrieval thread to obtain a target memory.

[0111] Shared memory allocation is performed on the index dump thread and the retrieval thread, so that both the index dump thread and the retrieval thread can access the same target memory, so that the retrieval thread based on the index structured data written by the index dump thread can be queried in real time.

[0112] S704: Content association processing is performed on the target memory with the index transfer thread and the retrieval thread respectively.

[0113] After determining the target memory, the target memory is content-associated with the index dump thread and the retrieval thread, respectively, so that the index dump thread and the retrieval thread are associated with each other. In this way, the retrieval thread can be determined based on the index dump thread. Of course, multiple index dump threads can correspond to one retrieval thread, multiple index dump threads can correspond to multiple retrieval threads, or one index dump thread can correspond to multiple retrieval threads. In addition, one index dump thread can correspond to one retrieval thread.

[0114] In an embodiment provided by the present application, after obtaining the index structured data from the first message queue in S604 and storing the index structured data in the target memory, the following steps are included:

[0115] The memory usage ratio of the target memory is detected. When the memory usage ratio is greater than or equal to a preset memory usage ratio threshold, the index structured data in the target memory is transferred to a target disk associated with the target memory, and the transferred index structured data in the target memory is cleaned up.

[0116] Detect the memory occupancy ratio of the target memory to obtain the memory occupancy ratio information of the target memory, and determine whether the memory occupancy ratio of the target memory is greater than or equal to the preset memory occupancy ratio threshold. The preset memory occupancy ratio threshold can be preset manually. When the memory occupancy ratio is greater than or equal to the preset memory occupancy ratio threshold, it indicates that the storage space of the target memory is insufficient. At this time, the index structured data in the target memory can be transferred to the target disk associated with the target memory, and the transferred index structured data in the target memory can be cleaned up to release the space of the target memory. It should be noted that when the retrieval thread accesses the target memory, it can access the target disk based on the built-in program in the target memory.

[0117] In a feasible implementation, qsrchd (Query Search Engine Database) can decide to periodically transfer the index structured data in the memory to the target disk associated with the target memory according to the memory usage ratio of the target memory, the amount of data, etc. qsrchd has an independent retrieval thread to provide retrieval, and an independent index transfer thread to obtain Protobuf data from qbus and then merge the index into the memory index.

[0118] In another embodiment provided by the present application, the index structured data in S202 is obtained by constructing index data by the index construction module on the original data, including:

[0119] Based on the index building module, the original data is preprocessed to obtain a forward index and an inverted index, and the forward index and the inverted index are structured to obtain index structured data.

[0120] Based on the index building module, the original data is preprocessed to perform forward construction to obtain a forward index and to perform inverted construction to obtain an inverted index. After the forward index and the inverted index are obtained, the forward index and the inverted index are structured to obtain index structured data.

[0121] See also Figure 8 , Figure 8 The following is a schematic diagram of the architecture of an index building module provided in an embodiment of the present application. Specifically, the index building module may be an Extractor, and the data format of the index building module supports json, xml, etc. Fig. 9Spout is a way to receive data, supporting reading local files, qbus message queues, and kafka message queues. qbus is a distributed message queue, and kafka is a high-throughput distributed publish-subscribe message system. Bolt is the actual index construction processing module, where modules_manager is used for module management and internal modules, feature_extr_manager is used for feature extraction, some features need to be extracted from other fields and regenerated; weight_calclt_manager is used to generate some weight values, such as word weights, which are required for sorting; forward_build_manager is a forward construction module used to build forward indexes; inverted_build_manager is an inverted construction module used to build inverted indexes; extractor_out_manager is an output module that supports output to local files, qbus, kafka, etc.; other modules include configuration management, word segmentation, basic conversion, synonyms, etc.

[0122] See also Fig. 9 , Fig. 9 A schematic diagram of the architecture of an index retrieval method corresponding to an embodiment of the present application.

[0123] First, the crawler crawls data from the Internet in real time, and pushes it directly to the screening module through the pipeline. The filtered data is directly written to the message queue qbus. Based on the Extractor, the original data is obtained from the qbus, and the data is pre-processed such as parsing, analysis, and word segmentation. Then, the forward index and inverted index are built, and the built forward index and inverted index are converted into Protobuf structure, and then the Protobuf structure is pushed to another qbus. The Extractor is deployed as a distributed service on the docker cluster, which realizes containerization and can be quickly and dynamically expanded. In the Extractor-based index construction, the document is pre-processed (word segmentation, special characters, etc.), and the forward index and inverted index are generated (both the forward index and the inverted index are independently built for one document or a small batch of documents). In the index writing phase: the indexes generated in the index construction phase are merged into one index, and the index files (disk or memory) are generated. These generated index files are used in the retrieval phase. During the database construction phase, the time consumed by the index construction and index writing phases differs greatly. The index construction accounts for 99% of the total time, while the time consumed by the index writing can be basically ignored. Based on this situation, the index construction phase can be separated out independently. Fig.10Qfed is the service component corresponding to the retrieval module, serving the retrieval control calculation thread and the attribute reading and summary thread. Index conversion in the index management thread is to convert data from the target memory to the target disk, dump means writing data to the target disk, and database change means clearing the data in the target memory. The alarm monitoring mechanism is used for data security monitoring, and the disaster recovery mechanism includes disaster recovery backup of remote servers.

[0124] In this embodiment, offline database building is implemented on the real-time engine, and the decoupling of index building and retrieval processes is achieved. The problem of index building will not affect the stability of the retrieval module. After offline, there is no problem of resource competition between the threads corresponding to index building and the retrieval threads. They can be expanded independently according to resource usage, avoiding resource waste. At the same time, containerization is implemented, which can be expanded quickly. All Extractors form a construction cluster, and each Extractor consumes a part of the data. When the amount of data to be built is relatively large, it can be dynamically increased, which solves the problems of slow single-machine construction and excessive data increase affecting timeliness. The data of a current machine can only be built locally, while this application can be built in the entire cluster.

[0125] An embodiment of the present application also provides a computer-readable storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing the steps of any method in the above embodiments.

[0126] See also Fig.10 , Fig.10 A schematic diagram of the structure of an index retrieval device provided in an embodiment of the present application.

[0127] The index retrieval device 1000 comprises:

[0128] The index building module 1010 is adapted to transfer the index structured data from the index building module to the target memory; wherein the index structured data is obtained by the index building module building the index data on the original data;

[0129] The retrieval module 1020 is suitable for performing index retrieval processing based on the index structured data in the target memory in a data retrieval scenario; wherein the index construction module and the retrieval module are independent of each other.

[0130] Optionally, the index building module 1010 includes:

[0131] A raw data acquisition submodule, adapted to acquire raw data using an index building module based on at least two reference containers in the container cluster;

[0132] The merging submodule is adapted to construct an index structure for the original data based on each reference container using the index building module, obtain distributed index structured data corresponding to each reference container, and merge the distributed index structured data to obtain index structured data.

[0133] Optionally, the index retrieval device 1000 further includes:

[0134] The deployment module is adapted to determine at least two reference containers in the container cluster and deploy the index construction module to the at least two reference containers.

[0135] Optionally, the deployment module includes:

[0136] A reference container deployment information determination submodule, adapted to obtain the index data volume of the original data, and determine the reference container deployment information from the container cluster based on the index data volume;

[0137] The reference container determination submodule is adapted to determine at least two reference containers from the container cluster based on the reference container deployment information.

[0138] Optionally, the raw data acquisition submodule includes:

[0139] An acquisition submodule, adapted to acquire original data and write the original data into a second message queue;

[0140] The calling submodule is adapted to call corresponding index building modules in at least two reference containers in the container cluster to obtain original data from the second message queue.

[0141] Optionally, the retrieval module 1020 includes:

[0142] A push submodule, adapted to call the index building module to push the index structured data in the index building module to the first message queue;

[0143] The storage submodule is adapted to obtain the index structured data from the first message queue and store the index structured data in the target memory.

[0144] Optionally, the index retrieval device 1000 includes:

[0145] A shared memory allocation submodule is adapted to perform shared memory allocation processing on the index transfer thread and the retrieval thread to obtain the target memory;

[0146] The content association submodule is adapted to perform content association processing on the target memory and the index transfer thread and the retrieval thread respectively.

[0147] Optionally, the index building module 1010 is adapted to control the index dump thread to dump the index structured data in the index building module into the target memory;

[0148] The retrieval module 1020 is suitable for performing index retrieval processing using the index structured data in the target memory based on the retrieval thread corresponding to the retrieval module in a data retrieval scenario; wherein the index transfer thread and the retrieval thread share the target memory.

[0149] Optionally, the index retrieval device 1000 further includes:

[0150] The memory occupancy ratio detection module is suitable for detecting the memory occupancy ratio of the target memory. When the memory occupancy ratio is greater than or equal to the preset memory occupancy ratio threshold, the index structured data in the target memory is transferred to the target disk associated with the target memory, and the transferred index structured data in the target memory is cleaned up.

[0151] Optionally, the index building module 1010 is further adapted to pre-process the original data to obtain a forward index and a reverse index, and perform structured processing on the forward index and the reverse index to obtain index structured data.

[0152] See also Fig.11 , Fig.11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.11 As shown, the electronic device 1100 may include: at least one processor 1101 , at least one network interface 1104 , a user interface 1103 , a memory 1105 , and at least one communication bus 1102 .

[0153] The communication bus 1102 is used to realize the connection and communication between these components.

[0154] The user interface 1103 may include a display screen (Display) and a camera (Camera), and the optional user interface 1103 may also include a standard wired interface and a wireless interface.

[0155] The network interface 1104 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0156] Among them, the processor 1101 may include one or more processing cores. The processor 1101 uses various interfaces and lines to connect various parts within the entire electronic device 1100, and executes various functions and processes data of the electronic device 1100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1105, and calling data stored in the memory 1105. Optionally, the processor 1101 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1101 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1101, and it can be implemented separately through a chip.

[0157] Among them, the memory 1105 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory, ROM). Optionally, the memory 1105 includes a non-transitory computer-readable storage medium. The memory 1105 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1105 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1105 may also be optionally at least one storage device located away from the aforementioned processor 1101. As Fig.11 As shown, the memory 1105 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an index retrieval program.

[0158] exist Fig.11In the electronic device 1100 shown, the user interface 1103 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1101 can be used to call the index retrieval program stored in the memory 1105 and specifically perform the following operations:

[0159] Transferring the index structured data from the index building module to the target memory; wherein the index structured data is obtained by the index building module building the index data on the original data;

[0160] In the data retrieval scenario, the index structured data in the target memory is used for index retrieval processing based on the retrieval module; wherein the index building module and the retrieval module are independent of each other.

[0161] Optionally, when the processor 1101 executes the index structured data to construct the index data by the index construction module on the original data, specifically:

[0162] Based on at least two reference containers in the container cluster, an index building module is used to obtain raw data;

[0163] Based on each reference container, an index construction module is used to construct an index structure for the original data to obtain distributed index structured data corresponding to each reference container, and the distributed index structured data are merged to obtain index structured data.

[0164] Optionally, before the processor 1101 acquires original data by using the index building module based on at least two reference containers in the container cluster, specifically performs:

[0165] At least two reference containers in the container cluster are determined, and the index building module is deployed to the at least two reference containers.

[0166] Optionally, when the processor 1101 determines at least two reference containers in the container cluster, specifically performs:

[0167] Obtain the index data volume of the original data, and determine the reference container deployment information from the container cluster based on the index data volume;

[0168] At least two reference containers are determined from the container cluster based on the reference container deployment information.

[0169] Optionally, when the processor 1101 acquires original data by using the index building module based on at least two reference containers in the container cluster, the processor 1101 specifically performs:

[0170] Obtaining original data, and writing the original data into the second message queue;

[0171] The corresponding index building modules in at least two reference containers in the container cluster are called to obtain original data from the second message queue.

[0172] Optionally, when the processor 1101 executes transferring the index structured data from the index building module to the target memory, the processor 1101 specifically executes:

[0173] Calling the index building module to push the index structured data in the index building module to the first message queue;

[0174] The index structured data is obtained from the first message queue, and the index structured data is stored in the target memory.

[0175] Optionally, the processor 1101 is further adapted to execute:

[0176] Performing shared memory allocation processing on the index transfer thread and the retrieval thread to obtain the target memory;

[0177] The target memory is content-associated with the index transfer thread and the retrieval thread respectively.

[0178] Optionally, when the processor 1101 executes transferring the index structured data from the index building module to the target memory, the processor 1101 specifically executes:

[0179] Controlling the index transfer thread based on the index building module to transfer the index structured data in the index building module to the target memory;

[0180] When the processor 1101 performs index retrieval processing based on the retrieval module using the index structured data in the target memory in the data retrieval scenario, the processor 1101 specifically performs:

[0181] In the data retrieval scenario, based on the retrieval thread corresponding to the retrieval module, the index structured data in the target memory is used to perform index retrieval processing; wherein the index transfer thread and the retrieval thread share the target memory.

[0182] Optionally, after the processor 1101 obtains the index structured data from the first message queue and stores the index structured data in the target memory, it specifically executes:

[0183] The memory usage ratio of the target memory is detected. When the memory usage ratio is greater than or equal to a preset memory usage ratio threshold, the index structured data in the target memory is transferred to a target disk associated with the target memory, and the transferred index structured data in the target memory is cleaned up.

[0184] Optionally, when the processor 1101 executes the index structured data to construct the index data by the index construction module on the original data, specifically:

[0185] Based on the index building module, the original data is preprocessed to obtain a forward index and an inverted index, and the forward index and the inverted index are structured to obtain index structured data.

[0186] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0187] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0189] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0190] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present application.

[0191] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0192] The above is a description of an index retrieval method, device, electronic device and computer-readable storage medium provided in an embodiment of the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.

Claims

1. An index retrieval method, wherein: The method comprises: Transferring the index structured data from the index building module to the target memory; wherein the index structured data is obtained by the index building module building the index data on the original data; In a data retrieval scenario, the index structured data in the target memory is used for index retrieval processing based on the retrieval module; wherein the index construction module and the retrieval module are independent of each other.

2. The method according to claim 1, wherein: The transferring of the index structured data from the index building module to the target memory includes: Calling the index building module to push the index structured data in the index building module to the first message queue; The index structured data is obtained from the first message queue, and the index structured data is stored in the target memory.

3. The method according to claim 1, wherein: The transferring of the index structured data from the index building module to the target memory includes: Controlling the index transfer thread based on the index building module to transfer the index structured data in the index building module to the target memory; In the data retrieval scenario, the index retrieval process is performed based on the retrieval module using the index structured data in the target memory, including: In a data retrieval scenario, based on a retrieval thread corresponding to a retrieval module, index structured data in the target memory is used to perform index retrieval processing; wherein the index transfer thread and the retrieval thread share the target memory.

4. The method according to claim 3, wherein: The method further comprises: Performing shared memory allocation processing on the index transfer thread and the retrieval thread to obtain a target memory; The target memory is respectively subjected to content association processing with the index transfer thread and the retrieval thread.

5. The method according to claim 2, wherein: The step of obtaining the index structured data from the first message queue and storing the index structured data in the target memory includes: Detect the memory occupancy ratio of the target memory. When the memory occupancy ratio is greater than or equal to a preset memory occupancy ratio threshold, transfer the index structured data in the target memory to a target disk associated with the target memory, and clean up the transferred index structured data in the target memory.

6. The method according to claim 1, wherein: The index structured data is obtained by the index building module constructing index data on the original data, including: Based on at least two reference containers in the container cluster, the index building module is used to obtain raw data; The index construction module is used to construct an index structure for the original data based on each reference container to obtain distributed index structured data corresponding to each reference container, and the distributed index structured data are merged to obtain index structured data.

7. The method according to claim 6, wherein: Before acquiring the original data by using the index building module based on at least two reference containers in the container cluster, the method further includes: At least two reference containers in the container cluster are determined, and the index construction module is deployed to the at least two reference containers.

8. An index retrieval device, wherein: The device comprises: An index building module, adapted to transfer the index structured data from the index building module to the target memory; wherein the index structured data is obtained by the index building module performing index data building on the original data; The retrieval module is suitable for performing index retrieval processing based on the index structured data in the target memory in a data retrieval scenario; wherein the index construction module and the retrieval module are independent of each other.

9. An electronic device, wherein: The electronic device includes: Processor; and A memory arranged to store computer executable instructions which, when executed, cause the processor to perform a method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, which, when executed by a processor, implement the method of any one of claims 1 to 7.

Images