Index updating method and device, electronic equipment and computer readable storage medium
By introducing index version levels in index updates and updating vector indexes only when necessary, the resource waste caused by full update of vector and text indexes is solved, and more efficient index updates are achieved.
Patent Information
- Application Number
- CN202311353398.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-18
- Publication Date
- 2025-04-18
AI Technical Summary
Vectors and text indexes are built uniformly during construction, and they need to be fully updated during updates, resulting in waste of resources, especially text indexes, which generally do not change.
By setting the index version level, determine the version level of the current vector index and historical vector index, and only update the vector index when the current version level is greater than the historical version level to avoid updating the text index.
It effectively avoids unnecessary updates to text indexes, reduces resource consumption, and improves the efficiency of index updates.
Smart Images
Figure CN119847988A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to an index updating method, device, electronic device and computer-readable storage medium. Background Art
[0002] With the continuous development of information technology, in order to facilitate the rapid search of data, data can be organized together in a specific format to build an index. For example, vector and text indexes are built in a unified manner when they are built, and they often need to be fully updated when they are updated. However, text indexes generally do not change, resulting in a large amount of resources being consumed when updating the index. Summary of the invention
[0003] The embodiments of the present application provide an index updating method, device, electronic device and computer-readable storage medium, which can solve the technical problem that vector and text indexes are constructed uniformly during construction, and the vector and text indexes often need to be fully updated during updating, while the text index generally does not change, resulting in a large amount of resources being consumed when updating the index.
[0004] In a first aspect, an embodiment of the present application provides an index updating method, the method comprising:
[0005] Obtaining a current index version level, and determining a current vector index corresponding to the current index version level;
[0006] Determine identification information corresponding to the current vector index, and determine a historical vector index corresponding to the current vector index and a historical index version level of the historical vector index based on the identification information;
[0007] When the current index version level is greater than the historical index version level, the historical vector index is updated based on the current vector index.
[0008] Optionally, the determining identification information corresponding to the current vector index includes:
[0009] Determine an index path of the current vector index, and search for a URL file corresponding to the current vector index based on the index path;
[0010] The identification information corresponding to the current vector index is determined based on the URL file.
[0011] Optionally, the determining, based on the identification information, a historical vector index corresponding to the current vector index and a historical index version level of the historical vector index includes:
[0012] Determine a URL file list based on the URL file corresponding to each current vector index;
[0013] The historical vector index data corresponding to the historical vector index is obtained, and the historical vector index data and the URL file list are matched according to the identification information to obtain the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index.
[0014] Optionally, determining the URL file list based on the URL file corresponding to each current vector index includes:
[0015] A URL file list is established based on the file name and index path of the URL file corresponding to each current vector index; the URL file includes URL information, identification information and storage information.
[0016] Optionally, matching the historical vector index data with the URL file list by using the identification information includes:
[0017] Constructing a target index for the URL file list and the identification information, and determining identification information of each historical vector index in the historical vector index data;
[0018] The identification information of each historical vector index is queried based on the target index.
[0019] Optionally, before obtaining the current index version level, the method further includes:
[0020] The current index information corresponding to the current vector index is stored separately;
[0021] The historical index information corresponding to the historical vector index is stored separately.
[0022] Optionally, the updating the historical vector index based on the current vector index includes:
[0023] The historical vector index is replaced based on the current vector index.
[0024] Optionally, the replacing the historical vector index based on the current vector index includes:
[0025] Determine current forward index information and current reverse index information based on the current vector index;
[0026] Determine historical forward index information and historical reverse index information based on the historical vector index;
[0027] The historical forward index information is replaced based on the current forward index information, and the historical inverted index information is replaced based on the current inverted index information.
[0028] In a second aspect, an embodiment of the present application provides an index updating device, the device comprising:
[0029] An acquisition module, adapted to acquire a current index version level and determine a current vector index corresponding to the current index version level;
[0030] a determination module, adapted to determine identification information corresponding to the current vector index, and determine a historical vector index corresponding to the current vector index and a historical index version level of the historical vector index based on the identification information;
[0031] The updating module is adapted to update the historical vector index based on the current vector index when the current index version level is greater than the historical index version level.
[0032] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:
[0033] Processor; and
[0034] A memory arranged to store computer executable instructions, which when executed cause the processor to perform any of the methods described above.
[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, implement any of the methods described above.
[0036] The beneficial effects brought by the technical solutions provided in some embodiments of the present application include at least: first, determining the current index version level, obtaining the current vector index corresponding to the current index version level, then determining the identification information corresponding to the current vector index, obtaining the historical vector index corresponding to the current vector index through the identification information, and determining the historical index version level corresponding to the historical vector index, then comparing the current index version level with the historical index version level, and when the current index version level is greater than the historical index version level, the current vector index corresponding to the current index version level updates the historical vector index. The present application sets the index version level for the vector index, and the corresponding vector index can be determined by using the index version level, and the current index version level can be compared with the historical index version level. After determining that the current index version level is a new version level, the historical vector index is updated through the current vector index, thereby avoiding the update of the text index, and solving the problem that the vector and text indexes are uniformly constructed when they are constructed, and the vector and text indexes often need to be fully updated when they are updated, but the text index generally does not change, resulting in a large amount of resources being consumed when updating the index. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0038] Figure 1 An exemplary system architecture diagram of an index updating method provided in an embodiment of the present application;
[0039] Figure 2 A flowchart of an index updating method provided in an embodiment of the present application;
[0040] Figure 3 A schematic diagram of a preprocessing process corresponding to an index updating method provided in an embodiment of the present application;
[0041] Figure 4 A schematic diagram of separately storing index information corresponding to a forward order and a reverse order provided in an embodiment of the present application;
[0042] Figure 5 A schematic diagram of replacing a historical vector index based on a current vector index provided in an embodiment of the present application;
[0043] Figure 6 A schematic diagram of an architecture for fast updating of vector indexes provided in an embodiment of the present application;
[0044] Figure 7 A schematic diagram of a process for determining identification information corresponding to a current vector index provided in an embodiment of the present application;
[0045] Figure 8 A schematic diagram of a process for determining a historical vector index corresponding to a current vector index and a historical index version level of the historical vector index provided in an embodiment of the present application;
[0046] Fig. 9 A schematic diagram of a process for matching historical vector index data with a URL file list provided in an embodiment of the present application;
[0047] Fig.10 A schematic diagram of the structure of an index updating device provided in an embodiment of the present application;
[0048] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] In order to make the features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the embodiments of the present application.
[0050] In the related art, in order to quickly build an index, vectors and texts are built uniformly when the index is built. Usually, vectors are built based on a vector construction model. Since the vector construction model changes frequently, the vectors often need to be fully updated, but the text usually does not change. However, when updating the index, since the vectors and texts are built uniformly when the index is built, it is necessary to update all the vector and text indexes. Because the amount of data is relatively large and the text index itself does not change, there is no point in updating it. At the same time, a large number of indexes are updated to the search machine, which is also a great challenge to the bandwidth. Therefore, there is an urgent need for a method to update only the vector index to solve the technical problem that the vector and text indexes are built uniformly when they are built, and the vector and text indexes often need to be fully updated when they are updated, but the text index generally does not change, resulting in a large amount of resources being consumed when updating the index.
[0051] In order to solve the above technical problems, an embodiment of the present application provides an index update method, which first determines the current index version level, obtains the current vector index corresponding to the current index version level, then determines the identification information corresponding to the current vector index, obtains the historical vector index corresponding to the current vector index through the identification information, and determines the historical index version level corresponding to the historical vector index, then compares the current index version level with the historical index version level, and when the current index version level is greater than the historical index version level, the current vector index corresponding to the current index version level updates the historical vector index. The embodiment of the present application sets the index version level for the vector index, and the corresponding vector index can be determined through the index version level, and can compare the current index version level with the historical index version level. After determining that the current index version level is a new version level, the historical vector index is updated through the current vector index, thereby avoiding the update of the text index, and solving the problem that the vector and text indexes are uniformly constructed when they are constructed, and the vector and text indexes often need to be fully updated when they are updated, but the text index generally does not change, resulting in a large amount of resources being consumed when updating the index.
[0052] See also Figure 1 , Figure 1An exemplary system architecture diagram of an index updating method provided in an embodiment of the present application.
[0053] like Figure 1 As shown, the system architecture may include an electronic device 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the electronic device 101 and the server 103. The network 102 may include various types of wired communication links or wireless communication links, for example, the wired communication link includes an optical fiber, a twisted pair, or a coaxial cable, and the wireless communication link includes a Bluetooth communication link, a Wireless-Fidelity (Wi-Fi) communication link, or a microwave communication link.
[0054] The electronic device 101 can interact with the server 103 through the network 102 to receive messages from the server 103 or send messages to the server 103, or the electronic device 101 can interact with the server 103 through the network 102 to receive messages or data sent by other users to the server 103. The electronic device 101 can be hardware or software. When the electronic device 101 is hardware, it can be various electronic devices, including but not limited to smart watches, smart phones, tablet computers, laptop portable computers and desktop computers. When the electronic device 101 is software, it can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules (for example: used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited here.
[0055] The server 103 may be a business server that provides various services. It should be noted that the server 103 may be hardware or software. When the server 103 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or it may be implemented as a single server. When the server 103 is software, it may be implemented as multiple software or software modules (for example, for providing distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.
[0056] In an embodiment of the present application, the server 103 can obtain the current index version level and determine the current vector index corresponding to the current index version level; then, determine the identification information corresponding to the current vector index, and determine the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index based on the identification information; finally, when the current index version level is greater than the historical index version level, the historical vector index is updated based on the current vector index.
[0057] It should be understood that Figure 1 The number of electronic devices, networks and servers in the figure is only illustrative, and any number of electronic devices, networks and servers may be used according to implementation requirements.
[0058] See also Figure 2 , Figure 2 A flowchart of an index update method provided for an embodiment of the present application. The execution subject of the embodiment of the present application can be a server that executes the index update method, or a processor in the server that executes the index update method, or an index update service in the server that executes the index update method. For the convenience of description, the specific execution process of the index update method is introduced below by taking the execution subject being a processor in the server as an example.
[0059] like Figure 2 As shown, the index update method includes:
[0060] S202: Obtain the current index version level, and determine the current vector index corresponding to the current index version level.
[0061] The current index version level may be determined based on the number of times the vector index is updated. For example, if the update cycle corresponding to the current vector index is the fourth update cycle, the current index version level may correspond to the fourth update cycle. The current index version level includes but is not limited to the index version number, etc. Specifically, the current index version level may be obtained in real time through the index management service.
[0062] Vectors can change continuously due to changes in their corresponding vector construction models. However, in order to avoid unnecessary waste of resources, the corresponding current vector index is not constructed after each vector update. In each update cycle corresponding to the vector index, the corresponding current vector index is constructed for the vector whose vector construction model has changed, and the current index version level corresponding to the current vector index is updated. There is no need to construct a corresponding vector index for the current vector whose vector construction model has not changed. At this time, the index version level of the vector index corresponding to the current vector whose vector construction model has not changed will not change. Among them, the vector can be a vectorized representation of an image, audio, text, and object features. Therefore, the index construction method corresponding to the embodiment of the present application includes but is not limited to constructing corresponding vectors for images, audio, text, and object features.
[0063] Each vector index has a corresponding index version level. When the vector corresponding to the vector index is updated and the new vector index is currently in the update cycle, the index version level corresponding to the vector index can be updated. Since each vector index has a corresponding index version level, after determining the current index version level, all current vector indexes corresponding to the current index version level can be pulled.
[0064] S204: Determine identification information corresponding to the current vector index, and determine the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index based on the identification information.
[0065] After obtaining the current vector index, the identification information corresponding to each current vector index is determined. Different current vector indexes correspond to different identification information. When the historical vector is updated to obtain the current vector, the historical vector index corresponding to the historical vector and the current vector index corresponding to the current vector have the same identification information. Specifically, the identification information may be docid, i.e., document ID (Identity document, ID number).
[0066] After determining the identification information corresponding to the current vector index, the identification information corresponding to the updated vector will not change due to the vector update caused by the change in the vector construction model, so the historical vector index corresponding to the current vector index can be determined through the identification information, and the historical index version level of the historical vector index can be obtained. It should be noted that the historical vector index corresponding to the current vector index determined based on the identification information is usually stored in a search engine, and the search engine is used to provide search services.
[0067] S206: When the current index version level is greater than the historical index version level, the historical vector index is updated based on the current vector index.
[0068] After obtaining the current index version level corresponding to the current vector index and the historical index version level corresponding to the historical vector index, the current index version level and the historical index version level are compared. Specifically, the version priority corresponding to the current index version level and the version priority corresponding to the historical index version level can be determined, and the version priority corresponding to the current index version level and the version priority corresponding to the historical index version level can be compared based on a pre-established version priority sequence table.
[0069] When the version priority corresponding to the current index version level is higher than the version priority corresponding to the historical index version level, it indicates that the current index version level is greater than the historical index version level. At this time, the historical vector index in the retrieval machine can be updated through the current vector index; when the version priority corresponding to the current index version level is lower than or equal to the version priority corresponding to the historical index version level, it indicates that the current index version level is less than or equal to the historical index version level. At this time, the historical vector index does not need to be updated.
[0070] The embodiment of the present application provides an index update method, first determining the current index version level, obtaining the current vector index corresponding to the current index version level, then determining the identification information corresponding to the current vector index, obtaining the historical vector index corresponding to the current vector index through the identification information, and determining the historical index version level corresponding to the historical vector index, then comparing the current index version level with the historical index version level, and when the current index version level is greater than the historical index version level, the current vector index corresponding to the current index version level updates the historical vector index. In addition, the embodiment of the present application completes the update of hundreds of billions of vector indexes, and the computing resource usage is reduced to 1 / 11 of the original, and the size of the index that needs to be updated is reduced to 1 / 11 of the original.
[0071] The embodiment of the present application sets an index version level for the vector index, and the corresponding vector index can be determined by the index version level. The current index version level and the historical index version level can be compared. After determining that the current index version level is a new version level, the historical vector index is updated by the current vector index, thereby avoiding the update of the text index. This solves the technical problem that the vector and text indexes are constructed uniformly during construction, but the vector and text indexes often need to be fully updated during update, while the text index generally does not change, resulting in a large amount of resources being consumed when updating the index.
[0072] See also Figure 3 , Figure 3 A flowchart of a preprocessing process corresponding to an index update method provided in an embodiment of the present application. In an embodiment provided in the present application, before obtaining the current index version level, the method further includes:
[0073] S302: Separately store current index information corresponding to the current vector index.
[0074] The current index information corresponding to the current vector index includes the vector index forward information and the vector index reverse information corresponding to the current vector index. Specifically, when the current index information corresponding to the current vector index is stored separately, the vector forward index information of the vectors corresponding to different fields can be stored separately.
[0075] like Figure 4 As shown, Figure 4 A schematic diagram of separately storing index information corresponding to the forward and reverse order provided in an embodiment of the present application. Figure 4 In the forward sorting diagram, each field corresponds to an independent vector forward sorting directory. The vector forward sorting directory may include vector forward sorting indexes of different index version levels, where the head part is used to identify the vector forward sorting index, and v stands for value, which represents the value corresponding to the key in the key value corresponding to the vector forward sorting index.
[0076] Similarly, when the current index information corresponding to the current vector index is stored separately, the vector inverted index information of the vectors corresponding to different fields can be stored separately. Figure 4 As shown in the inverted index diagram, each field corresponds to an independent vector inverted directory, and the vector inverted directory may include vector inverted indexes of different index version levels, wherein the head part is used to identify the vector inverted index, and doc represents the file information corresponding to the vector inverted index.
[0077] It should be noted that the vector forward index is the forward information from the URL (Uniform Resource Locator) to the term list, term weights and attributes, and term offsets generated after the web page is parsed, segmented, and relevance calculated. Its uses include: generating inverted indexes and making references for sorting during retrieval. The vector inverted index is the reverse information from the term to the URL zipper, the weight, attributes, and offset of the term in the URL generated by flipping the forward data.
[0078] S304: separately storing the historical index information corresponding to the historical vector index.
[0079] The historical index information corresponding to the historical vector index includes the vector index forward information and the vector index reverse information corresponding to the historical vector index. Specifically, when the historical index information corresponding to the historical vector index is stored separately, the vector forward index information of the vectors corresponding to different fields can be stored separately. Figure 4 As shown, in Figure 4 In the forward sorting diagram, each field corresponds to an independent vector forward sorting directory, and the vector forward sorting directory may include vector forward sorting indexes of different index version levels.
[0080] Similarly, when the historical index information corresponding to the historical vector index is stored separately, the vector inverted index information of the vectors corresponding to different fields can be stored separately. Figure 4 As shown, in the inverted index diagram, each field corresponds to an independent vector inverted directory, and the vector inverted directory may include vector inverted indexes of different index version levels.
[0081] In the embodiment of the present application, current index information corresponding to the current vector index is stored separately, and historical index information corresponding to the historical vector index is stored separately, so that the vector index is distinguished from the text index and stored separately, so that the vector index can be updated separately at the structural level.
[0082] In an embodiment provided in the present application, updating the historical vector index based on the current vector index in S206 includes:
[0083] The historical vector index is replaced based on the current vector index.
[0084] After determining the current vector index and the historical vector index corresponding to the current vector index in the search engine, determine the storage location of the historical vector index in the search engine, and replace the historical vector index with the current vector index. At the macro level, there are many ways to update the index, such as full database construction, incremental database construction, and rolling database construction.
[0085] Full database construction is to build an index for all data. Incremental database construction is to build the new data, modified data, and changed features into the full database every day. Because the full data is too much, the database construction cannot be completed in one day, so incremental database construction is required to build the new data into the full database to achieve data update. Rolling database construction is because the amount of data is huge (tens of billions of web pages), and it is difficult to complete the index construction in one day. A part of the index is built every day, and the full database construction is completed within a cycle. The embodiments provided in this application are mainly used for rolling database construction. Because the text index is built in a rolling manner, the text index will not change within a certain period of time.
[0086] For example, Figure 5 As shown, Figure 5 A schematic diagram of replacing a historical vector index based on a current vector index is provided in an embodiment of the present application. The above-mentioned replacement of a historical vector index based on a current vector index includes:
[0087] S502: Determine current forward index information and current reverse index information based on the current vector index.
[0088] After determining the current vector index, the current forward index information and the current inverted index information corresponding to the current vector index are determined, and the current forward index information and the current inverted index information corresponding to the same current vector index are matched with each other. When updating, the current forward index information and the current inverted index information that need to match each other are updated synchronously.
[0089] S504: Determine historical forward index information and historical reverse index information based on the historical vector index.
[0090] After determining the historical vector index, determine the historical forward index information and the historical inverted index information corresponding to the historical vector index. The historical forward index information and the historical inverted index information corresponding to the same historical vector index are mutually matched. When updating, the mutually matched historical forward index information and historical inverted index information need to be updated synchronously to provide subsequent retrieval services.
[0091] S506: Replace the historical forward index information based on the current forward index information, and replace the historical inverted index information based on the current inverted index information.
[0092] Since the current vector index and the historical vector index correspond to each other, after obtaining the current forward index information and the current inverted index information that match each other, and obtaining the historical forward index information and the historical inverted index information that match each other, the historical forward index information in the search engine can be replaced by the current forward index information, and the historical inverted index information in the search engine can be replaced by the current inverted index information.
[0093] Please refer to Figure 6 , Figure 6 A schematic diagram of a vector index fast update architecture provided in an embodiment of the present application. Figure 6 As shown, after obtaining the vector index corresponding to the current vector index, the historical forward index information and the historical inverted index information in the retrieval machine are replaced by the vector forward index and the vector inverted index in the vector index, that is, the current forward index information and the current inverted index information, thereby providing online retrieval services.
[0094] In the embodiment provided in the present application, the current forward index information replaces the historical forward index information, and the current reverse index information replaces the historical reverse index information, thereby updating the version levels of the historical forward index information and the historical reverse index information, and providing a retrieval service.
[0095] like Figure 7 As shown, Figure 7 A schematic diagram of a process for determining identification information corresponding to a current vector index provided in an embodiment of the present application. In an embodiment provided in the present application, determining identification information corresponding to a current vector index includes:
[0096] S702: Determine the index path of the current vector index, and search for a URL file corresponding to the current vector index based on the index path.
[0097] After obtaining the current vector index, the current vector index is parsed to determine the index path, and a search is performed based on the index path to find the URL file corresponding to the current vector index. The URL file may be a URL file, and the URL file may include URL information (such as URL), identification information (such as docid), and storage information (such as bucketid). Bucketid may be a bucket number. Specifically, since rolling database construction is to build a part of the index every day, the full database construction is completed within one cycle. Therefore, a part of the index built each time in the same cycle can be put into a bucket, and this bucket number is the corresponding storage information.
[0098] It should be noted that the embodiment of the present application can perform index construction in a general Map Reduce manner for processing and index construction of large-scale data sets. This allows multiple buckets to be constructed in parallel, and the degree of concurrency can be adjusted dynamically. For a single bucket vector, the forward information is constructed by multiple processing modules in parallel, and is divided according to docid. The reverse information of the vector is constructed by multiple processing modules in parallel, and can be divided according to the center point of the multi-layer index model. After the forward and reverse information are constructed, they can be placed under a path to form an actual index.
[0099] S704: Determine identification information corresponding to the current vector index based on the URL file.
[0100] After obtaining the URL file, the URL file is parsed to obtain identification information corresponding to the current vector index in the URL file.
[0101] The embodiment of the present application searches for a URL file corresponding to the current vector index through the index path of the current vector index, and finally determines the identification information corresponding to the current vector index through the URL file, thereby realizing fast search of the identification information.
[0102] like Figure 8 As shown, Figure 8 A schematic diagram of a process for determining a historical vector index corresponding to a current vector index and a historical index version level of the historical vector index provided in an embodiment of the present application. In S204, determining the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index based on the identification information includes:
[0103] S802: Determine a URL file list based on the URL file corresponding to each current vector index.
[0104] Wherein, a URL file list may be established based on the file name and index path of the URL file corresponding to each current vector index; the URL file includes URL information, identification information and storage information.
[0105] Specifically, when the website address file is a url file, each url file name and the index path corresponding to the url file can be put together to form a urllist file as a whole. The corresponding url in each bucket in the urllist file can be found.
[0106] S804: Obtain historical vector index data corresponding to the historical vector index, match the historical vector index data with the URL file list through identification information, and obtain the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index.
[0107] Since the URL file list is obtained based on the relevant data corresponding to the current vector index, and the historical vector index data is obtained based on the relevant data corresponding to the historical vector index, the historical vector index data and the URL file list can be matched based on the identification information, and then the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index can be filtered out from the historical vector index data and the URL file list. Specifically, Figure 6 As shown, data_1 (historical vector index data) and urllist can be used to intersect to obtain a matching result, and data in the format of bucketid, docid, and original vector can be generated based on the matching result and data_1 to further determine the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index.
[0108] In an embodiment of the present application, a URL file list is established through the URL file corresponding to each current vector index, and then the URL file list corresponding to the current vector index and the historical vector index data corresponding to the historical vector index are matched based on the identification information, so as to obtain the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index in batches and conveniently.
[0109] like Fig. 9 As shown, Fig. 9 A flow chart of matching historical vector index data and a URL file list provided in an embodiment of the present application. In an embodiment provided in the present application, matching historical vector index data and a URL file list by identification information in S804 includes:
[0110] S902: construct a target index for the URL file list and the identification information, and determine the identification information of each historical vector index in the historical vector index data.
[0111] Since the URL file list has a large amount of data, the requirements for matching the historical vector index data and the URL file list are high. Therefore, in order to achieve the matching of historical vector index data and URL file lists, this application proposes a high-performance kv service. The high-performance kv service is implemented based on dense_hash_map. The kv service is a project based on key-value storage, which usually builds a high-performance, scalable distributed storage system. The high-performance kv service can achieve batch query Qps (Queries per second) of up to 1000w / s. Specifically, a distributed index is constructed for the urllist and identification information. After the construction is completed, it is published to the high-performance kv cluster to obtain the target index. Then, the identification information of each historical vector index in the historical vector index data is determined to prepare for subsequent queries.
[0112] S904: query the identification information of each historical vector index based on the target index.
[0113] like Figure 6 As described above, after obtaining the target index, the data of data_1 can be read based on the Merge service. In the mapping stage, the key corresponding to md5 is used, where md5 is the encoding of the url, and the kv service is requested in batches based on the identification information corresponding to the key. After processing based on the kv service, the bucket number, docid and other information corresponding to the obtained md5 are obtained. According to the bucket number, it is mapped to the corresponding processing module, so that the data of each bucket can be mapped to several specified processing modules for processing, and the data of different buckets will not be mapped to the same processing module. In addition, the data output by the Merge service has realized the division of buckets, and the data format can be md5docid vector.
[0114] The embodiment of the present application constructs a target index for the URL file list and identification information, thereby accelerating the process of matching the historical vector index data with the URL file list and improving efficiency.
[0115] An embodiment of the present application further provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing the steps of any method in the above embodiments.
[0116] See also Fig.10 , Fig.10 A schematic diagram of the structure of an index updating device provided in an embodiment of the present application.
[0117] The index updating device 1000 comprises:
[0118] An acquisition module 1010 is adapted to acquire a current index version level and determine a current vector index corresponding to the current index version level;
[0119] A determination module 1020, adapted to determine identification information corresponding to a current vector index, and determine a historical vector index corresponding to the current vector index and a historical index version level of the historical vector index based on the identification information;
[0120] The updating module 1030 is adapted to update the historical vector index based on the current vector index when the current index version level is greater than the historical index version level.
[0121] Optionally, the determining module 1020 includes:
[0122] A first determining unit, adapted to determine an index path of a current vector index, and search for a URL file corresponding to the current vector index based on the index path;
[0123] The second determining unit is adapted to determine identification information corresponding to the current vector index based on the URL file.
[0124] Optionally, the determining module 1020 further includes:
[0125] A URL file list determining unit, adapted to determine a URL file list based on the URL file corresponding to each current vector index;
[0126] The acquisition unit is adapted to acquire historical vector index data corresponding to the historical vector index, match the historical vector index data with the URL file list through identification information, and obtain the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index.
[0127] Optionally, the URL file list determination unit is adapted to establish a URL file list based on the file name and index path of the URL file corresponding to each current vector index; the URL file includes URL information, identification information and storage information.
[0128] Optionally, the acquiring unit includes:
[0129] The identification information determination subunit is adapted to construct a target index for the URL file list and the identification information, and determine the identification information of each historical vector index in the historical vector index data;
[0130] The query subunit is adapted to query the identification information of each historical vector index based on the target index.
[0131] Optionally, the index updating device 1000 further includes:
[0132] A first storage unit, adapted to separately store current index information corresponding to a current vector index;
[0133] The second storage unit is adapted to separately store the historical index information corresponding to the historical vector index.
[0134] Optionally, the updating module 1030 is adapted to replace the historical vector index based on the current vector index.
[0135] Optionally, the updating module 1030 includes:
[0136] A current vector index determining unit, adapted to determine current forward index information and current reverse index information based on the current vector index;
[0137] A historical vector index determining unit, adapted to determine historical forward index information and historical reverse index information based on the historical vector index;
[0138] The replacement unit is adapted to replace the historical forward index information based on the current forward index information, and to replace the historical inverted index information based on the current inverted index information.
[0139] See also Fig.11 , Fig.11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.11 As shown, the electronic device 1100 may include: at least one processor 1101 , at least one network interface 1104 , a user interface 1103 , a memory 1105 , and at least one communication bus 1102 .
[0140] The communication bus 1102 is used to realize the connection and communication between these components.
[0141] The user interface 1103 may include a display screen (Display) and a camera (Camera), and the optional user interface 1103 may also include a standard wired interface and a wireless interface.
[0142] The network interface 1104 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).
[0143] Among them, the processor 1101 may include one or more processing cores. The processor 1101 uses various interfaces and lines to connect various parts within the entire electronic device 1100, and executes various functions and processes data of the electronic device 1100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1105, and calling data stored in the memory 1105. Optionally, the processor 1101 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1101 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1101, and it can be implemented separately through a chip.
[0144] Among them, the memory 1105 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory, ROM). Optionally, the memory 1105 includes a non-transitory computer-readable storage medium. The memory 1105 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1105 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1105 may also be optionally at least one storage device located away from the aforementioned processor 1101. As Fig.11 As shown, the memory 1105 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an index update program.
[0145] exist Fig.11 In the electronic device 1100 shown, the user interface 1103 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1101 can be used to call the index update program stored in the memory 1105 and specifically perform the following operations:
[0146] Get the current index version level and determine the current vector index corresponding to the current index version level;
[0147] Determine identification information corresponding to the current vector index, and determine the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index based on the identification information;
[0148] When the current index version level is greater than the historical index version level, the historical vector index is updated based on the current vector index.
[0149] In some embodiments, when the processor 1101 determines the identification information corresponding to the current vector index, it specifically performs:
[0150] Determine the index path of the current vector index, and search for the URL file corresponding to the current vector index based on the index path;
[0151] The identification information corresponding to the current vector index is determined based on the URL file.
[0152] In some embodiments, when the processor 1101 determines the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index based on the identification information, the processor 1101 specifically performs:
[0153] Determine a URL file list based on the URL file corresponding to each current vector index;
[0154] The historical vector index data corresponding to the historical vector index is obtained, and the historical vector index data and the URL file list are matched through the identification information to obtain the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index.
[0155] In some embodiments, when the processor 1101 determines the URL file list based on the URL file corresponding to each current vector index, it specifically executes:
[0156] A URL file list is established based on the file name and index path of the URL file corresponding to each current vector index; the URL file includes URL information, identification information and storage information.
[0157] In some embodiments, when the processor 1101 matches the historical vector index data with the URL file list through the identification information, it specifically performs:
[0158] Constructing a target index for the URL file list and the identification information, and determining the identification information of each historical vector index in the historical vector index data;
[0159] The identification information of each historical vector index is queried based on the target index.
[0160] In some embodiments, before obtaining the current index version level, the processor 1101 specifically executes:
[0161] The current index information corresponding to the current vector index is stored separately;
[0162] The historical index information corresponding to the historical vector index is stored separately.
[0163] In some embodiments, when the processor 1101 updates the historical vector index based on the current vector index, it specifically executes:
[0164] The historical vector index is replaced based on the current vector index.
[0165] In some embodiments, when the processor 1101 performs replacement processing on the historical vector index based on the current vector index, it specifically performs:
[0166] Determine current forward index information and current reverse index information based on the current vector index;
[0167] Determine historical forward index information and historical reverse index information based on the historical vector index;
[0168] The historical forward index information is replaced based on the current forward index information, and the historical inverted index information is replaced based on the current inverted index information.
[0169] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.
[0170] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0171] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.
[0172] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
[0173] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present application.
[0174] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0175] The above is a description of an index update method, device, electronic device and computer-readable storage medium provided in an embodiment of the present application. For technicians in this field, based on the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.
Claims
1. An index updating method, wherein: The method comprises: Obtaining a current index version level, and determining a current vector index corresponding to the current index version level; Determine identification information corresponding to the current vector index, and determine a historical vector index corresponding to the current vector index and a historical index version level of the historical vector index based on the identification information; When the current index version level is greater than the historical index version level, the historical vector index is updated based on the current vector index.
2. The method according to claim 1, wherein: The determining the identification information corresponding to the current vector index includes: Determine an index path of the current vector index, and search for a URL file corresponding to the current vector index based on the index path; The identification information corresponding to the current vector index is determined based on the URL file.
3. The method according to claim 2, wherein: The determining, based on the identification information, the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index includes: Determine a URL file list based on the URL file corresponding to each current vector index; The historical vector index data corresponding to the historical vector index is obtained, and the historical vector index data and the URL file list are matched according to the identification information to obtain the historical vector index corresponding to the current vector index and the historical index version level of the historical vector index.
4. The method according to claim 3, wherein: The determining of the URL file list based on the URL file corresponding to each current vector index includes: A URL file list is established based on the file name and index path of the URL file corresponding to each current vector index; the URL file includes URL information, identification information and storage information.
5. The method according to claim 3, wherein: The matching of the historical vector index data and the URL file list by using the identification information includes: Constructing a target index for the URL file list and the identification information, and determining identification information of each historical vector index in the historical vector index data; The identification information of each historical vector index is queried based on the target index.
6. The method according to claim 1, wherein: Before obtaining the current index version level, the method further includes: The current index information corresponding to the current vector index is stored separately; The historical index information corresponding to the historical vector index is stored separately.
7. The method according to claim 1, wherein: The updating the historical vector index based on the current vector index includes: The historical vector index is replaced based on the current vector index.
8. An index updating device, wherein: The device comprises: An acquisition module, adapted to acquire a current index version level and determine a current vector index corresponding to the current index version level; a determination module, adapted to determine identification information corresponding to the current vector index, and determine a historical vector index corresponding to the current vector index and a historical index version level of the historical vector index based on the identification information; The updating module is adapted to update the historical vector index based on the current vector index when the current index version level is greater than the historical index version level.
9. An electronic device, wherein: The electronic device includes: Processor; and A memory arranged to store computer executable instructions which, when executed, cause the processor to perform a method according to any one of claims 1 to 7.
10. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, which, when executed by a processor, implement the method of any one of claims 1 to 7.