Database construction method and apparatus, and storage medium

Abstract

This application discloses a database construction method, apparatus, and storage medium, which can be applied to various scenarios such as cloud technology, artificial intelligence, intelligent transportation, and the Internet of Vehicles. The method includes: determining a target document set based on the business indicator value corresponding to each document in a preset document set, and performing a first sort on the documents in the preset document set to obtain a first sorting result; determining the initial document number corresponding to each document in the preset document set based on the first sorting result; performing a second sort on each target document in the target document set based on the keyword sequence corresponding to each target document to obtain a second sorting result; determining the rearranged document number of each target document in the target document set based on the second sorting result; and constructing an inverted index database based on the initial document number corresponding to the non-target document and the rearranged document number corresponding to each target document in the target document set. This application improves the accuracy and efficiency of document querying.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a database construction method, apparatus and storage medium. Background Technology

[0002] Inverted indexes are a widely used and important technology in general search engines. Compared with other indexing formats, they strike a balance between search effectiveness and performance, as well as time and space consumption. The process of a search engine using an inverted index to process queries mainly involves: 1. Building an inverted index on a collection of documents; 2. The user inputs a query; 3. The search engine retrieves the inverted index and obtains the intersection of candidate documents for each search term; 4. Calculating document scores and returning the Top N results to the user. For search engines, each document must be assigned a unique numerical number during the inverted index building process for internal processing. The most common practice is to assign a number to each document in a certain order, such as chronological order or random order, from smallest to largest.

[0003] For general search engines, to speed up retrieval and improve user experience, they often don't fully search the entire index for a user's search request. Once the search engine has retrieved a certain number of documents, it may stop searching and return the currently retrieved results. Therefore, documents with smaller document IDs in the inverted index have a higher probability of being retrieved, while documents with later IDs may not be retrieved because the retrieval process has ended prematurely. In the process of numbering documents using existing algorithms, multiple documents with the same feature value are usually randomly numbered. However, there are still differences between these documents with the same feature value, making it impossible to maximize the grouping of similar documents in this set, thus failing to achieve the maximum compression rate of document IDs and reducing the accuracy and efficiency of document queries. Summary of the Invention

[0004] This application provides a database construction method, apparatus, and storage medium that can improve the accuracy and efficiency of document retrieval.

[0005] On the one hand, this application provides a database construction method, the method comprising:

[0006] Based on the business indicator value corresponding to each document in the preset document set, a target document set is determined and the documents in the preset document set are sorted for the first time to obtain a first sorting result; the target document set includes at least two target documents with the same business indicator value in the preset document set;

[0007] Based on the first sorting result, determine the initial document number corresponding to each document in the preset document set;

[0008] Based on the keyword sequence corresponding to each target document in the target document set, the target documents in the target document set are sorted a second time to obtain a second sorting result; the keyword sequence corresponding to each target document is determined based on the keywords corresponding to each target document.

[0009] Based on the second sorting result, determine the rearranged document number of each target document in the target document set;

[0010] An inverted index database is constructed based on the initial document numbers corresponding to non-target documents in the preset document set and the rearranged document numbers corresponding to each target document in the target document set.

[0011] On the other hand, a database construction apparatus is provided, the apparatus comprising:

[0012] The first sorting module is used to determine the target document set based on the business indicator value corresponding to each document in the preset document set and to sort the documents in the preset document set for the first time to obtain the first sorting result; the target document set includes at least two target documents in the preset document set with the same business indicator value;

[0013] The initial document number determination module is used to determine the initial document number corresponding to each document in the preset document set based on the first sorting result.

[0014] The second sorting module is used to sort each target document in the target document set a second time according to the keyword sequence corresponding to each target document in the target document set, and obtain a second sorting result; the keyword sequence corresponding to each target document is determined based on the keywords corresponding to each target document.

[0015] The rearranged document number determination module is used to determine the rearranged document number of each target document in the target document set based on the second sorting result;

[0016] The database construction module is used to construct an inverted index database based on the initial document numbers corresponding to non-target documents in the preset document set and the rearranged document numbers corresponding to each target document in the target document set.

[0017] On the other hand, a database construction device is provided, the device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the database construction method as described above.

[0018] On the other hand, a computer storage medium is provided that stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the database construction method described above.

[0019] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the database construction method as described above.

[0020] The database construction method, apparatus, and storage medium provided in this application have the following technical advantages:

[0021] This application determines a target document set based on the business indicator value corresponding to each document in a preset document set, and performs a first sorting on the documents in the preset document set to obtain a first sorting result. The target document set includes at least two target documents with the same business indicator value in the preset document set. Based on the first sorting result, an initial document number corresponding to each document in the preset document set is determined. Based on the keyword sequence corresponding to each target document in the target document set, a second sorting is performed on each target document in the target document set to obtain a second sorting result. The keyword sequence corresponding to each target document is determined based on the keywords corresponding to each target document. Based on the second sorting result, a rearranged document number for each target document in the target document set is determined. An inverted index database is constructed based on the initial document number corresponding to non-target documents in the preset document set and the rearranged document number corresponding to each target document in the target document set. For target document sets with the same business indicator value, this application re-sorts the initial document numbers of target documents in the target document set based on the keyword sequence corresponding to each target document, thereby reducing the distance between document numbers of similar documents and improving the accuracy and efficiency of document queries. Attached Figure Description

[0022] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a database construction system provided in an embodiment of this application;

[0024] Figure 2 This is a flowchart illustrating a database construction method provided in an embodiment of this application;

[0025] Figure 3 This is a flowchart illustrating a method for constructing a preset database according to an embodiment of this application;

[0026] Figure 4 This is a flowchart illustrating a method for reordering the initial document numbers of each target document in the target document set to obtain a reordered document number set, as provided in an embodiment of this application.

[0027] Figure 5 This is a flowchart illustrating a method for generating a keyword sequence for each target document based on the keywords corresponding to each target document in the target document set, according to an embodiment of this application.

[0028] Figure 6 This is a flowchart illustrating a method for determining the keywords of each target document in the target document set to obtain a keyword set, provided in an embodiment of this application.

[0029] Figure 7 This is a flowchart illustrating a method for generating a keyword sequence for each target document based on the category information of the keywords corresponding to each target document, as provided in an embodiment of this application.

[0030] Figure 8 This is a schematic diagram of a document retrieval method provided in an embodiment of this application;

[0031] Figure 9 This is a schematic diagram showing the changes in word spacing and compression rate before and after reordering, provided in an embodiment of this application.

[0032] Figure 10 This is a flowchart of a method for secondary sorting of a target document set provided in an embodiment of this application;

[0033] Figure 11 This is a schematic diagram of the structure of a database construction apparatus provided in an embodiment of this application;

[0034] Figure 12 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0037] Please see Figure 1 , Figure 1 This is a schematic diagram of a database construction system provided in an embodiment of this application, such as... Figure 1 As shown, the database construction system may include at least server 01 and client 02.

[0038] Specifically, in this embodiment, server 01 may include a standalone server, a distributed server, or a server cluster composed of multiple servers. It may also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 01 may include network communication units, processors, and memory, etc. Specifically, server 01 can be used to build an inverted index database and search for a set of documents to be queried within the inverted index database.

[0039] Specifically, in this embodiment, the client 02 may include physical devices such as smartphones, desktop computers, tablets, laptops, digital assistants, smart wearable devices, smart speakers, in-vehicle terminals, and smart TVs. It may also include software running on the physical device, such as web pages provided to users by service providers, or applications provided to users by those service providers. Specifically, the client 02 can be used to display a set of documents to be queried based on document query information.

[0040] The following describes a database construction method proposed in this application. Figure 2This is a flowchart illustrating a database construction method provided in an embodiment of this application. This specification provides the operational steps of the method described in the embodiments or flowchart, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0041] Specific examples Figure 2 As shown, the method may include:

[0042] S201: Based on the business indicator value corresponding to each document in the preset document set, determine the target document set and perform a first sorting on the documents in the preset document set to obtain a first sorting result; the target document set includes at least two target documents with the same business indicator value in the preset document set.

[0043] In this embodiment of the application, before determining the target document set based on the business indicator value corresponding to each document in the preset document set and performing a first sorting on the documents in the preset document set to obtain the first sorting result, the method further includes:

[0044] Obtain the preset document set, which includes at least two documents;

[0045] In this embodiment of the application, the preset document set may include the text data, title, etc. of the documents.

[0046] In this embodiment, each search engine can set different business metrics. These metrics may include document click-through rate, number of views, etc., or comprehensive metrics determined based on document category, creation time, text length, etc. The business metric value represents the importance of a document; a higher value indicates a more important document, allowing for a smaller document ID to be prioritized in the search process. Conversely, a lower value indicates less important documents, requiring a larger document ID to ensure that documents with higher importance and smaller IDs are prioritized in the search. This allows the search engine to provide users with high-quality documents based on their document query requests, improving the user experience.

[0047] In this embodiment, the business metric value can be a feature value of each document determined by the PageRank algorithm, and the feature value can characterize the importance of the document. The basic idea of ​​the PageRank algorithm is to define a random walk model, i.e., a first-order Markov chain, on a directed graph to describe the behavior of a random walker randomly visiting each node along the directed graph. Under certain conditions, in the extreme case, the probability of visiting each node converges to a stationary distribution. At this time, the stationary probability value of each node is its PageRank value, representing the importance of the node. PageRank is recursively defined, and the calculation of PageRank can be performed through an iterative algorithm. The PageRank algorithm assigns a relatively coarse-grained rating to documents, resulting in a large number of sets with the same document rating. These sets are generally randomly sorted, which cannot maximize the distribution of similar documents and thus cannot achieve the maximum compression ratio.

[0048] In this embodiment of the application, a target document set is determined based on the business indicator value corresponding to each document in the preset document set, and the documents in the preset document set are sorted for the first time to obtain a first sorting result, including:

[0049] Documents with the same business indicator values ​​in the preset document set are selected as target documents to obtain the target document set;

[0050] Based on the business indicator value corresponding to each document in the preset document set, a first document set with a business indicator value greater than the target document and a second document set with a business indicator value less than the target document are determined.

[0051] Based on the business indicator values ​​corresponding to each first document in the first document set, the first documents are sorted to obtain the first document sorting result;

[0052] Determine the target document sorting result for each target document in the target document set;

[0053] In this embodiment of the application, the target document sorting result excludes the first and last target documents in the sorting order, and the adjacent documents of the other target documents are all documents in the target document set.

[0054] Specifically, in this embodiment of the application, since the business indicator values ​​corresponding to each target document in the target document set are the same, the target documents in the target document set can be randomly sorted.

[0055] Specifically, in this embodiment of the application, determining the target document sorting result of each target document in the target document set includes:

[0056] Obtain the generation time of each target document in the target document set;

[0057] Based on the generation time of each target document, the target documents in the target document set are sorted to obtain the target document sorting result.

[0058] In this embodiment of the application, the target documents can be sorted according to their relevant attribute information. For example, the target sorting result can be obtained based on the generation time of each target document.

[0059] Based on the business indicator values ​​corresponding to each second document in the second document set, the second documents are sorted to obtain the second document sorting result;

[0060] The first sorting result is determined based on the first document sorting result, the target document sorting result, and the second document sorting result.

[0061] S203: Based on the first sorting result, determine the initial document number corresponding to each document in the preset document set.

[0062] In this embodiment of the application, determining the initial document number corresponding to each document in the preset document set based on the first sorting result includes:

[0063] Based on the sorting result corresponding to each first document in the first document set, the first number of each first document is determined to obtain the first number set;

[0064] Based on the sorting result corresponding to each target document in the target document set and the first number set, the target document number of each target document in the target document set is determined, and the target document number set is obtained.

[0065] Specifically, in this embodiment, the maximum number in the first number set can be obtained, and the target number of each target document can be determined based on the maximum number and the sorting result corresponding to each target document. The target document numbers of each target document in the target document set are consecutive numbers, which can be randomly numbered according to the random sorting of the target documents, or numbered according to the relevant attribute information of the target documents; for example, for target documents sorted according to their generation time, smaller target numbers can be assigned to target documents with later generation times, and larger target numbers can be assigned to target documents with earlier generation times.

[0066] Based on the sorting result corresponding to each second document in the second document set and the target document number set, a second number for each second document is determined, thus obtaining a second number set.

[0067] Specifically, in this embodiment of the application, the maximum target number in the target document number set can be obtained, and the second number of each second document can be determined according to the sorting result of the maximum target number and each second document.

[0068] The numbers in the first number set, the target document number set, and the second number set are used to determine the initial document number.

[0069] In this embodiment of the application, for documents with the same business indicator value in the preset document set, if the document number is set by random sorting or sorting according to the document generation time, it is impossible to maximize the arrangement of similar document sets and obtain the maximum compression rate of the inverted index database.

[0070] In this embodiment of the application, in order to maximize the arrangement of similar documents and improve the data compression rate of the inverted index database, the target documents in the target document set with the same business indicator value can be reordered twice, thereby arranging similar documents in a concentrated manner and improving the data compression rate of the inverted index database.

[0071] S205: Based on the keyword sequence corresponding to each target document in the target document set, perform a second sorting on each target document in the target document set to obtain a second sorting result; the keyword sequence corresponding to each target document is determined based on the keywords corresponding to each target document.

[0072] In this embodiment of the application, the step of performing a second sorting on each target document in the target document set based on the keyword sequence corresponding to each target document in the target document set to obtain the second sorting result includes:

[0073] Obtain the first and second values ​​of the keyword sequences corresponding to any two target documents in the target document set, and use them as the current values ​​corresponding to the two target documents.

[0074] Compare the current values ​​corresponding to any two target documents; the two target documents include the first target document and the second target document.

[0075] If the current value of the first target document is greater than the current value of the second target document, it is determined that the first target document is ranked before the second target document.

[0076] In this embodiment of the application, the method further includes:

[0077] If the current values ​​corresponding to any two target documents are the same, obtain the value following the current value in the keyword sequence corresponding to the two target documents, and use it as the current value corresponding to each of the two target documents.

[0078] Jump to the step of comparing the current values ​​corresponding to any two target documents.

[0079] In this embodiment, the keyword sequence comparison method is as follows: starting from position number 1, the values ​​of the two keyword sequences at each position number are compared sequentially to position number N. If the values ​​are the same, the next position is compared; if the values ​​are different, the sequence with the larger value is considered the larger value. Through the above comparison method, a second order based on the keyword sequences is obtained from largest to smallest. If the keyword sequences corresponding to two target documents are the same, it indicates that the two target documents are similar, and the two target documents are then randomly sorted.

[0080] In this embodiment of the application, the number of rearranged document numbers is consistent with the number of document numbers in the initial document number set corresponding to the target document set, and the number of document numbers in the rearranged document number set is consistent with the number of document numbers in the initial document number set corresponding to the target document set.

[0081] In a specific embodiment, as shown in Table 1, Table 1 presents the secondary sorting results of the document set in Table 3. Documents 3 and 5 have the same keyword sequence, so their order can be swapped. For identical keyword sequences, random sorting is possible. Table 2 presents the re-sorting results of the document set in Table 4. It can be seen that the document numbers for all documents in the target document set still correspond to the initial document numbers 1-7; only some document numbers have been swapped, reducing the distance between document numbers of similar documents.

[0082] Table 1

[0083] Document Number document Keywords Keyword sequence 1 Document 1 A, B, C, D, E 11111 2 Document 3 A, B, C, D 11110 3 Document 5 A, B, C, D, G 11110 4 Document 7 A, B, C 11100 5 Document 2 A, B 11000 6 Document 6 A, E 10001 7 Document 4 F 00000

[0084] Table 2

[0085] Document Number document Keywords Keyword sequence 1 Document 5 A, B, C, D, G 32210 2 Document 2 A, B 31000 3 Document 1 A, B, C, D, E 21111 4 Document 3 A, B, C, D 12110 5 Document 7 A, B, C 11100 6 Document 6 A, E 10001 7 Document 4 F 00000

[0086] In a specific embodiment, such as Figure 10 As shown, Figure 10 Here is a flowchart of a method for secondary sorting of a target document set, the method comprising:

[0087] (1) Sort the document collection in descending order based on the PageRank algorithm feature values ​​of the documents to obtain the first order. The PageRank algorithm sorts documents according to their importance. By using the PageRank algorithm, users can see important and high-quality web pages first when searching using a search engine, thus making it easier to obtain the information they need.

[0088] (2) Count the keywords in each document in the document set to obtain the frequency of occurrence of the keyword set in the document set. If a keyword appears in a document, increment the frequency of that keyword by one. For example, if a keyword appears in the keyword list of n documents, the frequency of that keyword is recorded as n. Sort the keywords in descending order according to their frequency, and extract the top N keywords by frequency. We record this list of N ordered keywords as TopN keywords. Generate corresponding position numbers 1-N for the TopN keywords. The keyword with the highest frequency has a position number of 1, the keyword with the second highest frequency has a position number of 2, and so on, with the keyword with the Nth highest frequency having a position number of N.

[0089] (3) Generate a keyword sequence corresponding to the TopN keywords for each document: If a keyword of a document is in the TopN keyword set, then in the keyword sequence of the document, the value of the position number k corresponding to the keyword is assigned to 1; the other positions in the keyword sequence are assigned to 0.

[0090] (4) Perform a secondary sort on the keyword sequences of the documents. If the first PageRank feature values ​​of two documents are the same, obtain the second order based on the comparison of their keyword sequences. The keyword sequence comparison method is as follows: starting from position number 1, compare the values ​​of the two keyword sequences at each position number. If the values ​​are the same, compare the next position; if the values ​​are different, the sequence with the value 1 is the larger value. Through the above comparison method, sort from largest to smallest to obtain the second order based on the keyword sequences. Sets of documents with the same first and second orders are randomly sorted.

[0091] (5) The new document numbers are reassigned according to the final sorting of the documents in the target document set, and the resorting is completed.

[0092] Traditional methods, after the first sort, randomly sort documents with the same value, failing to guarantee that similar documents will have similar document numbering orders. This embodiment's secondary sorting ensures similar documents have similar document numbering orders. Keyword classification only requires simple statistical methods, resulting in low algorithm complexity, no complex calculations, and low resource consumption. The keyword classification process only needs to count the frequency of keywords across all documents, obtaining the N most frequent keywords after sorting. The value of N can be adjusted based on the document set size, resource constraints, and changes in compression ratio after re-sorting, offering high flexibility. The generated keyword sequence can be compressed and saved using a bitmap method, consuming minimal resources. Furthermore, the comparison algorithm required for the secondary sorting is low in complexity and easy to implement.

[0093] In this embodiment, the target documents in the target document set are sorted twice to avoid random sorting and to increase the distance between document numbers of similar target documents.

[0094] In the embodiments of this application, such as Figure 3 As shown, before performing a second sorting of the target documents in the target document set based on the keyword sequence corresponding to each target document in the target document set to obtain the second sorting result, the method further includes:

[0095] S2041: Obtain the keywords corresponding to each target document in the target document set;

[0096] S2043: Generate a keyword sequence for each target document based on the keywords corresponding to each target document in the target document set.

[0097] In this embodiment, keywords corresponding to each target document in the target document set can be extracted to generate a keyword sequence for each target document, thereby reordering the target documents in the target document set according to the sequence number. The generated keyword sequence can be compressed and saved using a bitmap method, which consumes few resources. A bitmap, also known as raster graphics or raster image, is an image represented using a pixel array (dot-matrix).

[0098] In the embodiments of this application, such as Figure 4 As shown, generating a keyword sequence for each target document based on the keywords corresponding to each target document in the target document set includes:

[0099] S20431: Determine the keywords of each target document in the target document set to obtain a keyword set;

[0100] In this embodiment, document keywords are a highly refined summary of the document content and theme. Two different documents with the same or similar keywords are more likely to express similar themes and content. Keywords can be extracted from each target document to obtain the keyword set corresponding to the target document set. Each target document can correspond to one or more keywords.

[0101] In this embodiment of the application, determining the keywords of each target document in the target document set to obtain a keyword set includes:

[0102] Keywords are extracted from the text corresponding to each target document to obtain the keyword set.

[0103] In this embodiment of the application, a keyword set can be constructed based on the text keywords of the target document, and the target documents in the target document set can be reordered.

[0104] In the embodiments of this application, such as Figure 5 As shown, the step of determining the keywords of each target document in the target document set to obtain a keyword set includes:

[0105] S204311: Obtain the text, anchor points, and breadcrumbs corresponding to each target document in the target document set;

[0106] In this embodiment, anchors are a type of hyperlink used in web page creation, also called named anchors. Like a quick locator, they are a type of hyperlink within a page and are widely used. Named anchors are used to set markers within a document, typically placed at specific topics or at the top of the document. Links to these named anchors can then be created, quickly taking visitors to their designated locations. Creating links to named anchors involves two steps: first, creating the named anchor, and then creating links to that anchor. The concept of breadcrumb navigation comes from the fairy tale "Hansel and Gretel." When Hansel and Gretel were walking through the forest, they got lost, but they found breadcrumbs scattered along their path, which helped them find their way home. Therefore, breadcrumb navigation tells visitors their location on the website and how to return. Keywords can be extracted based on the target document's corresponding text, anchors, and breadcrumbs, thus bringing the numbering of similar documents closer together and improving the accuracy of document numbering.

[0107] S204313: Extract keywords from the text corresponding to each target document to obtain the first keyword corresponding to each target document;

[0108] S204315: Extract keywords from the anchor points corresponding to each target document to obtain the second keyword corresponding to each target document;

[0109] S204317: Extract keywords from the breadcrumbs corresponding to each target document to obtain the third keyword corresponding to each target document;

[0110] S204319: The first keyword, the second keyword, and the third keyword corresponding to each target document are determined as the keywords of each target document;

[0111] In the embodiments of this application, the first keyword, the second keyword, and the third keyword may be the same or different.

[0112] In this embodiment of the application, the first keyword, the second keyword, and the third keyword of the target document can all be used as keywords of the target document.

[0113] S2043111: The set of keywords of each target document in the target document set is determined as the keyword set.

[0114] In this embodiment of the application, a keyword set can be determined based on the first keyword, the second keyword, and the third keyword of the target document.

[0115] In this embodiment, document keywords are highly refined and summarized document content and themes. Two different documents with the same or similar keywords are more likely to express similar themes and content. Keywords that appear more frequently in the entire document set are likely to be more important and commonly used, and are more likely to be used in the search engine retrieval process. By centrally sorting the document numbers in the inverted index of a word to minimize the spacing between document numbers, and in conjunction with the index compression encoding algorithm, higher compression rates and decompression efficiency can be achieved. Keywords that appear more frequently in the entire document set may have longer corresponding index inverted indexes, and the improvement in compression rate after reordering the inverted index is more significant.

[0116] S20433: Determine the frequency of occurrence of each keyword in the keyword set;

[0117] In this embodiment of the application, determining the frequency of occurrence of each keyword in the keyword set includes:

[0118] Determine the number of target documents corresponding to each keyword;

[0119] The number of target documents corresponding to each keyword is taken as the frequency of occurrence of each keyword.

[0120] In this embodiment, for a keyword set constructed using only text keywords, the target document corresponding to a keyword is defined as the document whose text contains the keyword; thus, the keywords in the keyword set are categorized. If a keyword appears in a document, its frequency is incremented by one. For example, if a keyword appears in a keyword list of n documents, its frequency is recorded as n. Based on the descending order of the frequency of each keyword after statistical analysis, the top N keywords are extracted, and this list of N ordered keywords is recorded as TopN keywords, i.e., the first category of keywords; all other non-TopN keywords are determined as the second category of keywords.

[0121] In a specific embodiment, as shown in Table 3, Table 3 is a table showing the correspondence between the initial document number and the document number and text keywords of a target document set.

[0122] Table 3

[0123] Document Number document Text keywords 1 Document 1 A, B, C, D, E 2 Document 2 A, B 3 Document 3 A, B, C, D 4 Document 4 F 5 Document 5 A, B, C, D, G 6 Document 6 A, E 7 Document 7 A, B, C

[0124] In this embodiment of the application, determining the frequency of occurrence of each keyword in the keyword set includes:

[0125] Based on the target document corresponding to each keyword, determine the number of texts, anchor points, and breadcrumbs corresponding to each keyword;

[0126] The sum of the number of texts, anchor points, and breadcrumbs corresponding to each keyword is determined as the frequency of occurrence for each keyword.

[0127] In this embodiment, the frequency of occurrence of each keyword can be determined based on the number of texts, anchor points, and breadcrumbs corresponding to each keyword, thereby reducing the distance between document numbers of similar documents and improving the accuracy of document sorting.

[0128] In a specific embodiment, as shown in Table 4, Table 4 is a table showing the correspondence between the initial document number and the document number, text keywords, anchor keywords, and breadcrumb keywords of a target document set.

[0129] Table 4

[0130] Document Number document Keywords Anchor point Breadcrumbs 1 Document 1 A, B, C, D, E A 2 Document 2 A, B A A 3 Document 3 A, B, C, D B 4 Document 4 F 5 Document 5 A, B, C, D, G A, B, C A 6 Document 6 A, E 7 Document 7 A, B, C

[0131] S20435: Determine the target keywords based on the frequency of occurrence of each keyword;

[0132] In this embodiment of the application, determining the target keyword based on the frequency of occurrence of each keyword includes:

[0133] Sort each keyword in the keyword set according to its frequency of occurrence from smallest to largest to obtain the keyword ranking result;

[0134] Based on the keyword ranking results, the top-ranked keywords are determined as the target keywords.

[0135] In this embodiment of the application, determining the target keyword based on the frequency of occurrence of each keyword includes:

[0136] Sort each keyword in the keyword set according to its frequency of occurrence from highest to lowest to obtain the keyword ranking result;

[0137] Based on the keyword ranking results, the first number of keywords ranked last are determined as the target keywords.

[0138] In this embodiment, if a keyword appears in a keyword list of m documents, the frequency of that keyword is denoted as m. Based on the descending order of the frequency of each keyword after statistical analysis, the top N keywords by frequency are obtained, and this list of N ordered keywords is denoted as TopN keywords, i.e., target keywords. N can be set according to actual conditions, for example, N can be 3, 10, etc.; there can be at least one target keyword.

[0139] In this embodiment of the application, determining the target keyword based on the frequency of occurrence of each keyword includes:

[0140] Keywords that appear more frequently than a preset frequency threshold are identified as target keywords.

[0141] In this embodiment of the application, a preset frequency threshold can be set to determine the target keywords, and keywords with higher frequency of occurrence can be used as target keywords.

[0142] In this embodiment of the application, the method further includes:

[0143] Based on the first quantity, determine the number of digits in the keyword sequence;

[0144] Based on the sorting results corresponding to the first number of target keywords, the position number corresponding to each target keyword is determined; the position number represents the position of the assigned value of the target keyword in the keyword sequence.

[0145] In this embodiment of the application, the number of digits in the keyword sequence can be determined based on the first number of target keywords; for example, if there are N target keywords, the keyword sequence can be determined to be N digits.

[0146] In the embodiments of this application, as shown in Table 5, Table 5 shows the frequency of occurrence of each target keyword in Table 3 and its position number in the keyword sequence; as shown in Table 6, Table 6 shows the frequency of occurrence of each target keyword in Table 4 and its position number in the keyword sequence.

[0147] Table 5

[0148] Keywords Frequency of occurrence Position number A 6 1 B 5 2 C 4 3 D 3 4 E 2 5

[0149] Table 6

[0150]

[0151]

[0152] S20437: Generate a keyword sequence for each target document based on the keywords corresponding to each target document and the target keywords.

[0153] In the embodiments of this application, such as Figure 6 As shown, generating a keyword sequence for each target document based on the keywords corresponding to each target document and the target keywords includes:

[0154] S204371: Assign a first value to the target keyword;

[0155] S204373: Generate a keyword sequence for each target document based on the keywords corresponding to each target document, the position number corresponding to each target keyword, and the first numerical value.

[0156] In the embodiments of this application, such as Figure 7 As shown, generating a keyword sequence for each target document based on the keywords corresponding to each target document, the position number corresponding to each target keyword, and the first numerical value includes:

[0157] S2043731: Use the keywords corresponding to any target document as preset keywords to obtain a preset keyword set;

[0158] S2043733: Determine the number of target keywords in the preset keyword set to obtain a second number;

[0159] S2043735: If the second quantity is less than the first quantity, determine the blank position in the target keyword sequence according to the position number corresponding to each target keyword in the preset keyword set and the number of digits in the keyword sequence; the target keyword sequence is the keyword sequence corresponding to any target document; the blank position is the position where no target keyword is assigned, determined based on the number of digits in the target keyword sequence.

[0160] In this embodiment of the application, the method may further include:

[0161] If the preset keyword set includes target keywords, determine the position number corresponding to each target keyword in the preset keyword set.

[0162] In this embodiment of the application, corresponding position numbers 1-N can be generated for TopN keywords (target keywords). The keyword with the highest frequency is assigned position number 1, the keyword with the second highest frequency is assigned position number 2, and so on, with the keyword with the Nth highest frequency being assigned position number N.

[0163] In this embodiment of the application, the method may further include:

[0164] If the second quantity equals the first quantity, the keyword sequence of each target document is generated based on the position number corresponding to each target keyword in the preset keyword set and the number of digits in the keyword sequence.

[0165] In this embodiment of the application, the first quantity and the second quantity are the same, indicating that there are no blank positions in the keyword sequence. At this time, the keyword sequence can be determined directly based on the position and value of the target keyword corresponding to each target document.

[0166] In this embodiment of the application, generating a keyword sequence for each target document based on the keywords corresponding to each target document and the target keywords includes:

[0167] Obtain the keywords corresponding to any target document to get a preset keyword set;

[0168] Determine the third number of preset keywords in the preset keyword set;

[0169] If the third quantity is greater than the first quantity, extract the first quantity of preset keywords from the preset keywords to obtain a filtered keyword set; the filtered keyword set includes keywords whose frequency of occurrence is greater than a preset value among the preset keywords.

[0170] If the set of filtered keywords includes non-target keywords, the non-target keywords are assigned a second value.

[0171] Based on the assigned values ​​and position numbers of each filter keyword in the filter keyword set, a keyword sequence corresponding to any target document is generated.

[0172] In this embodiment of the application, all keywords in each target document can be extracted, and a second number of keywords to be extracted can be determined. If the second number is greater than the first number, the first number of keywords can be further extracted, thereby ensuring that the number of bits in the keyword sequence is fixed, which facilitates subsequent comparison of keyword sequences of different target documents, thereby determining the document number.

[0173] S2043737: Assign a second value to the blank positions in the target keyword sequence;

[0174] In this embodiment, the first value and the second value are different. The first value can be set to the value 1 and the second value can be set to the value 0; or the first value and the second value can be set to any two different other numbers, and the first value is not 0.

[0175] S2043739: Generate the target keyword sequence based on the position number corresponding to the target keyword in the preset keyword set, the first value, and the second value.

[0176] In this embodiment of the application, if the target keyword appears in at least two of the text, anchor points, and breadcrumbs corresponding to the target document, the method for determining the value assigned to the target keyword includes:

[0177] Determine the frequency of occurrence of target keywords in the text, anchors, and breadcrumbs of the same target document;

[0178] If the target keyword exists in any two of the text, anchors, and breadcrumbs corresponding to the same target document, then the value corresponding to the target keyword is determined to be twice the first value;

[0179] If the target keyword exists in the text, anchors, and breadcrumbs of the same target document, then the value corresponding to the target keyword is determined to be three times the first value.

[0180] In this embodiment of the application, if the first value is 1, and the first type of keyword A exists in the text and anchor points corresponding to the same target document, then the target keyword A is assigned a value of 2 in the corresponding keyword sequence; if the target keyword A exists in the text, anchor points and breadcrumbs corresponding to the same target document, then the target keyword A is assigned a value of 3 in the corresponding keyword sequence.

[0181] In a specific embodiment, as shown in Table 7, Table 7 is the keyword sequence of each document in the document set corresponding to Table 3. The keyword sequence of the document is generated based on the position number of the keyword in Table 5 and the keyword corresponding to each document. As shown in Table 8, Table 8 is the keyword sequence of each document in the document set corresponding to Table 4. The keyword sequence of the document is generated based on the position number of the keyword in Table 6 and the target keyword in the text, anchors, and breadcrumbs corresponding to each document.

[0182] Table 7

[0183] document Text keywords Keyword sequence Document 1 A, B, C, D, E 11111 Document 2 A, B 11000 Document 3 A, B, C, D 11110 Document 4 F 00000 Document 5 A, B, C, D, G 11110 Document 6 A, E 10001 Document 7 A, B, C 11100

[0184] Table 8

[0185]

[0186]

[0187] S207: Based on the second sorting result, determine the rearranged document number of each target document in the target document set.

[0188] In this embodiment of the application, determining the rearranged document number of each target document in the target document set based on the second sorting result includes:

[0189] Obtain the initial document number corresponding to each target document in the target document set to obtain the initial target document number set;

[0190] Based on the second sorting result, the numbers in the initial target document number set are redistributed to each target document in the target document set to obtain the rearranged document number of each target document in the target document set; wherein, the rearranged document number of the first target document is less than the rearranged document number of the second target document.

[0191] S209: Construct an inverted index database based on the initial document numbers corresponding to the non-target documents in the preset document set and the rearranged document numbers corresponding to each target document in the target document set.

[0192] In this embodiment of the application, the step of constructing an inverted index database based on the initial document numbers corresponding to non-target documents in the preset document set and the rearranged document numbers corresponding to each target document in the target document set includes:

[0193] Each document in the preset document set is parsed to obtain the preset words corresponding to each document;

[0194] Based on the preset words and document number corresponding to each document, the correspondence between the preset words and document numbers is determined; the document number includes the initial document number and the rearranged document number.

[0195] The inverted index database is constructed based on the preset correspondence between words and document numbers.

[0196] In this embodiment, the document numbers corresponding to preset words can be compressed, thereby effectively reducing the size of the inverted index database. A general inverted index compression algorithm is as follows: 1. Calculate the N-1 intervals of the sorted N document numbers.<d1,d2...,dN-1> 2. Record the first document number (POS), and compress and save the N-1 document number intervals using a compression algorithm. During decompression, the complete inverted index document numbers are obtained based on the first document number (POS) and the decompressed interval list. For index compression technology, the smaller the intervals and the more concentrated the arrangement of the data to be compressed, the higher the compression ratio and the higher the decompression efficiency, thereby improving document query efficiency.

[0197] In this embodiment, if the target document set includes seven documents with document numbers 1-7, for the preset word A, before reordering, its corresponding document numbers are 1, 3, and 7, and after reordering, its corresponding document numbers are 1, 2, and 3. Obviously, after reordering, the spacing between the document numbers corresponding to the preset word A is reduced. This embodiment greatly improves the compression rate of the inverted index without affecting the retrieval effect, at the cost of a small increase in time and space consumption when creating the index, and effectively reduces the time and space consumption of decompressing the inverted index during retrieval.

[0198] Specifically, in this embodiment of the application, taking the compression of inverted indexes using the Bitpacking algorithm as an example, the Bitpacking algorithm treats every 64 numbers as a block for compression. For example... Figure 9 As shown, Figure 9 The diagram illustrates the changes in word spacing and compression ratio before and after reordering. Using the reordering algorithm in this embodiment, the largest spacing among the first 64 numbers is 16, requiring 4 bits for each number, totaling 256 bits for the compressed content. After reordering, the largest spacing is reduced to 4, requiring only 2 bits for each number, totaling 128 bits for the compressed content. Using the reordering method in this embodiment effectively improves the compression ratio of the inverted chain of word A. Correspondingly, due to the improved compression ratio, the I / O read size required for decompressing word A is reduced from 256 bits to 128 bits, thus improving decompression efficiency.

[0199] The evaluation criterion for inverted index compression is the compression ratio, which is the ratio of the compressed inverted index to the uncompressed inverted index. Table 9 shows a comparison of the compression ratios for a web search dataset using the simple PageRank sorting method and the secondary sorting method of this embodiment. The inverted index compression algorithm used is the bitpacking compression algorithm. The compression ratio using the simple PageRank sorting method is 26.23%, while the compression ratio using the method of this embodiment is 23.58%, representing a 2.65% improvement in compression ratio compared to the simple PageRank method.

[0200] Table 9

[0201] Inverted index compression ratio PageRank sorting method 26.23％ The method in this embodiment 23.58％

[0202] Table 10 shows the proportion of changes in the size of each inverted chain in the inverted index after compression of a web search dataset using the method of this embodiment. The results show that the proportion of inverted chains that become smaller after compression is greater than the proportion that become larger, which also explains the increase in the inverted index compression ratio in Table 9.

[0203] Table 10

[0204] Reduced size after compression No change in size after compression Enlarged after compression 64％ 1％ 35％

[0205] Table 11 shows the throughput changes when using this dataset for decompression. Compared with the method before reordering, the secondary reordering method in this embodiment has a significant improvement in decompression throughput, with an improvement rate of approximately 12%. It can be seen that this embodiment improves the document query efficiency.

[0206] Table 11

[0207] Throughput PageRank sorting method 6.5 million integers / second The method in this embodiment 7.28 million integers / second

[0208] Table 12 shows the data quantiles and corresponding recall ratios when retrieving a web search dataset using a specific instance request set. A data quantile is a numerical point that divides the probability distribution range of a random variable into several equal parts. This embodiment shows a significant improvement in the recall ratio at each data quantile compared to the PageRank method alone; compared to the PageRank method alone, this embodiment can recall high-quality results more quickly.

[0209] Table 12

[0210]

[0211]

[0212] In this embodiment of the application, the method further includes:

[0213] Based on the document query information, the set of document IDs that match the document query information is searched from the inverted index database.

[0214] In this embodiment, the document query information may include, but is not limited to, search terms, search statements, text, etc.; the document may be a webpage in a search engine; the method of this embodiment can be applied to document search and query in a search engine.

[0215] In this embodiment of the application, the step of searching for a set of document IDs matching the document query information from the inverted index database based on the document query information includes:

[0216] The document query information is parsed to obtain the parsed words;

[0217] In this embodiment of the application, the parsed words can be one or more.

[0218] Based on the parsed words, search the inverted index database for a set of document IDs that match the parsed words.

[0219] In this embodiment, the inverted index database can store the correspondence between preset words and document numbers; one preset word can correspond to one or more document numbers. The document numbers in the set of documents to be queried can be sorted in ascending order. For the search engine, during the process of building the inverted index, each document must be assigned a unique numerical number to facilitate internal processing.

[0220] In a specific embodiment, as shown in Table 13, Table 13 is a document set, including the correspondence between document numbers and document data; where document data includes document ID and document title; Table 14 is the inverted index database corresponding to the document set in Table 13; this database stores the correspondence between each word in the document and the document number; it also sets a word ID for each word; when parsing document query information, the corresponding word ID can be determined based on the parsed words, and then the corresponding document number can be found. During data storage, by compressing the numerical numbers corresponding to the documents in the inverted index, the file size of the inverted index can be effectively reduced. A general inverted index compression algorithm is as follows: 1. Calculate the N-1 intervals of the N document numbers in the sorted inverted table.<d1,d2...,dN-1> 2. Record the first document number (POS), and compress and save the N-1 document number intervals using a compression algorithm. The value of N can be adjusted based on the document set size, resource constraints, and changes in compression ratio after reordering, offering high flexibility. During decompression, the complete inverted index document numbers are obtained based on the first document number and the decompressed interval list. For index compression technology, the smaller the intervals and the more concentrated the arrangement of the data to be compressed, the higher the compression ratio and the higher the decompression efficiency. Therefore, reordering the document numbers, centrally sorting the document numbers in the inverted index to minimize intervals, and combining this with the index compression encoding algorithm can achieve higher compression ratios and decompression efficiency.

[0221] Table 13

[0222]

[0223] Table 14

[0224]

[0225]

[0226] In this embodiment of the application, the method further includes:

[0227] Based on the document query information, a set of candidate document IDs matching the document query information is searched from the inverted index database; the candidate document IDs in the candidate document ID set are sorted in order according to the size of their respective document IDs.

[0228] From the candidate document number set, a preset number of candidate document numbers are selected to obtain the document number set to be queried; the document number set to be queried includes the preset number of candidate document numbers that are sorted at the beginning or end of the candidate document number set;

[0229] The set of documents to be queried is determined based on the set of document IDs.

[0230] In this embodiment of the application, a candidate document number set can be determined based on an inverted index database; each candidate document number in the candidate document number set is sorted in ascending or descending order according to its corresponding document number.

[0231] From the candidate document number set, a preset number of candidate document numbers are selected to obtain the document number set to be queried; the document number set to be queried includes the preset number of candidate document numbers that are ranked first or last in the candidate document number set.

[0232] In this embodiment of the application, in order to speed up the search and improve the user experience, the entire index is usually not searched for a user's search request. A preset number of candidate document numbers can be selected from the candidate document number set. The preset number can be set according to the actual situation.

[0233] In a specific embodiment, such as Figure 8 As shown, Figure 8 This is a schematic diagram of a document retrieval method. When the search engine retrieves a certain number N, it will stop the retrieval and return the currently retrieved results. Therefore, documents with smaller document numbers in the inverted index have a higher probability of being retrieved. Since the retrieval process has ended early, documents with later numbers will not be recalled. This can improve retrieval speed and enhance user experience.

[0234] As can be seen from the technical solutions provided in the above embodiments of this application, the embodiments of this application determine a target document set based on the business indicator value corresponding to each document in a preset document set and perform a first sorting on the documents in the preset document set to obtain a first sorting result; the target document set includes at least two target documents with the same business indicator value in the preset document set; based on the first sorting result, the initial document number corresponding to each document in the preset document set is determined; based on the keyword sequence corresponding to each target document in the target document set, each target document in the target document set is sorted a second time to obtain a second sorting result; the keyword sequence corresponding to each target document is determined based on the keywords corresponding to each target document; based on the second sorting result, the rearranged document number of each target document in the target document set is determined; based on the initial document number corresponding to the non-target document in the preset document set and the rearranged document number corresponding to each target document in the target document set, an inverted index database is constructed. For target document sets with the same business indicator value, this application re-sorts the initial document numbers of the target documents in the target document set based on the keywords corresponding to each target document, thereby reducing the distance between document numbers of similar documents and improving the accuracy and efficiency of document query.

[0235] This application also provides a database construction apparatus, such as... Figure 11 As shown, the device includes:

[0236] The first sorting module 1110 is used to determine a target document set based on the business indicator value corresponding to each document in the preset document set and to sort the documents in the preset document set for the first time to obtain a first sorting result; the target document set includes at least two target documents in the preset document set with the same business indicator value;

[0237] The initial document number determination module 1120 is used to determine the initial document number corresponding to each document in the preset document set based on the first sorting result.

[0238] The second sorting module 1130 is used to sort each target document in the target document set a second time according to the keyword sequence corresponding to each target document in the target document set, and obtain a second sorting result; the keyword sequence corresponding to each target document is determined based on the keywords corresponding to each target document.

[0239] The rearranged document number determination module 1140 is used to determine the rearranged document number of each target document in the target document set based on the second sorting result;

[0240] The database construction module 1150 is used to construct an inverted index database based on the initial document number corresponding to the non-target document in the preset document set and the rearranged document number corresponding to each target document in the target document set.

[0241] In some embodiments, the apparatus may further include:

[0242] The keyword acquisition module is used to acquire the keywords corresponding to each target document in the target document set;

[0243] The keyword sequence generation module is used to generate a keyword sequence for each target document based on the keywords corresponding to each target document in the target document set.

[0244] In some embodiments, the keyword sequence generation module includes:

[0245] The keyword set determination submodule is used to determine the keywords of each target document in the target document set to obtain the keyword set;

[0246] The frequency of occurrence determination submodule is used to determine the frequency of occurrence of each keyword in the keyword set;

[0247] The target keyword determination submodule is used to determine the target keywords based on the frequency of occurrence of each keyword.

[0248] The keyword sequence generation submodule is used to generate a keyword sequence for each target document based on the keywords corresponding to each target document and the target keywords.

[0249] In some embodiments, the target keyword determination submodule includes:

[0250] The sorting unit is used to sort each keyword in the keyword set from smallest to largest according to the frequency of occurrence of each keyword, so as to obtain the keyword sorting result;

[0251] The target keyword determination unit is used to determine the first number of keywords that rank first in the keyword ranking as the target keywords based on the keyword ranking results.

[0252] In some embodiments, the apparatus further includes:

[0253] A number digit determination module is used to determine the number of digits in the keyword sequence based on the first quantity;

[0254] The position number determination module is used to determine the position number corresponding to each target keyword based on the sorting results corresponding to the first number of target keywords; the position number represents the position of the assigned value of the target keyword in the keyword sequence.

[0255] In some embodiments, the keyword sequence generation submodule includes:

[0256] The first assignment unit is used to assign a first numerical value to the target keyword;

[0257] The keyword sequence generation unit is used to generate a keyword sequence for each target document based on the keywords corresponding to each target document, the position number corresponding to each target keyword, and the first value.

[0258] In some embodiments, the keyword sequence generation unit includes:

[0259] The preset keyword set determination sub-unit is used to take the keywords corresponding to any target document as preset keywords to obtain the preset keyword set;

[0260] The second quantity determination subunit is used to determine the quantity of target keywords in the preset keyword set to obtain the second quantity;

[0261] The blank position determination subunit is used to determine the blank position in the target keyword sequence based on the position number corresponding to each target keyword in the preset keyword set and the number of digits in the keyword sequence if the second quantity is less than the first quantity; the target keyword sequence is the keyword sequence corresponding to any target document; the blank position is the position where no target keyword is assigned, determined based on the number of digits in the target keyword sequence.

[0262] The second assignment subunit is used to assign a second value to the blank positions in the target keyword sequence;

[0263] The target keyword sequence generation subunit is used to generate the target keyword sequence based on the position number corresponding to the target keyword in the preset keyword set, the first value, and the second value.

[0264] In some embodiments, the second sorting module includes:

[0265] The current value determination submodule is used to obtain the first and second values ​​of the keyword sequences corresponding to any two target documents in the target document set, and use them as the current values ​​corresponding to the two target documents.

[0266] The comparison submodule is used to compare the current values ​​corresponding to any two target documents; the two target documents include a first target document and a second target document.

[0267] The sorting submodule is used to determine that the sorting of the first target document is before that of the second target document if the current value of the first target document is greater than the current value of the second target document.

[0268] In some embodiments, the document numbering rearrangement determination module includes:

[0269] The initial target document number set acquisition submodule is used to acquire the initial document number corresponding to each target document in the target document set, thereby obtaining the initial target document number set;

[0270] The numbering allocation submodule is used to reallocate the numbers in the initial target document numbering set to each target document in the target document set according to the second sorting result, so as to obtain the rearranged document number of each target document in the target document set; wherein, the rearranged document number of the first target document is less than the rearranged document number of the second target document.

[0271] In some embodiments, the keyword set determination submodule includes:

[0272] The keyword set determination unit is used to extract keywords from the text corresponding to each target document to obtain the keyword set.

[0273] In some embodiments, the frequency determination submodule includes:

[0274] The target document quantity determination unit is used to determine the number of target documents corresponding to each keyword;

[0275] The first frequency determination unit is used to take the number of target documents corresponding to each keyword as the frequency of occurrence of each keyword.

[0276] In some embodiments, the keyword set determination submodule includes:

[0277] The text acquisition unit is used to acquire the text, anchor points, and breadcrumbs corresponding to each target document in the target document set;

[0278] The first keyword extraction unit is used to extract keywords from the text corresponding to each target document to obtain the first keyword corresponding to each target document.

[0279] The first keyword extraction unit is used to extract keywords from the anchor points corresponding to each target document to obtain the second keyword corresponding to each target document.

[0280] The third keyword extraction unit is used to extract keywords from the breadcrumbs corresponding to each target document to obtain the third keyword corresponding to each target document.

[0281] The keyword determination unit is used to determine the first keyword, the second keyword, and the third keyword corresponding to each target document as the keywords of each target document;

[0282] The keyword set determination unit is used to determine the set of keywords of each target document in the target document set as the keyword set.

[0283] In some embodiments, the frequency determination submodule includes:

[0284] The quantity determination unit is used to determine the number of texts, anchor points, and breadcrumbs corresponding to each keyword based on the target document corresponding to each keyword.

[0285] The second frequency determination unit is used to determine the frequency of occurrence of each keyword by summing the number of texts, anchor points, and breadcrumbs corresponding to each keyword.

[0286] In some embodiments, the apparatus further includes:

[0287] The candidate document ID set lookup module is used to search for a set of candidate document IDs that match the document query information from the inverted index database based on the document query information; the candidate document IDs in the candidate document ID set are sorted in order according to the size of their respective document IDs;

[0288] The document ID set determination module is used to filter out a preset number of candidate document IDs from the candidate document ID set to obtain the document ID set to be queried; the document ID set to be queried includes the preset number of candidate document IDs that are sorted at the beginning or end of the candidate document ID set;

[0289] The document set determination module is used to determine the document set to be queried based on the document number set to be queried.

[0290] The apparatus and method embodiments described herein are based on the same inventive concept.

[0291] This application provides a database construction device, which includes a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the database construction method provided in the above method embodiments.

[0292] Embodiments of this application also provide a computer storage medium, which can be disposed in a terminal to store at least one instruction or at least one program related to implementing a database construction method in the method embodiments. The at least one instruction or at least one program is loaded and executed by the processor to implement the database construction method provided in the above method embodiments.

[0293] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the database construction method provided in the above-described method embodiments.

[0294] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0295] The memory described in this application embodiment can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for the functions, etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0296] The database construction method embodiments provided in this application can be executed on mobile terminals, computer terminals, servers, or similar computing devices. Taking running on a server as an example, Figure 12 This is a hardware structure block diagram of a server for a database construction method provided in an embodiment of this application. For example... Figure 12As shown, the server 1200 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1210 (CPUs 1210 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 1230 for storing data, and one or more storage media 1220 (e.g., one or more mass storage devices) for storing application programs 1223 or data 1222. The memory 1230 and storage media 1220 may be temporary or persistent storage. The program stored in the storage media 1220 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 1210 may be configured to communicate with the storage media 1220 and execute the series of instruction operations stored in the storage media 1220 on the server 1200. Server 1200 may also include one or more power supplies 1260, one or more wired or wireless network interfaces 1250, one or more input / output interfaces 1240, and / or one or more operating systems 1221, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0297] The input / output interface 1240 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 1200. In one example, the input / output interface 1240 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 1240 may be a radio frequency (RF) module for wireless communication with the Internet.

[0298] Those skilled in the art will understand that Figure 12 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 1200 may also include... Figure 12 The more or fewer components shown, or having the same Figure 12 The different configurations shown.

[0299] As can be seen from the embodiments of the database construction method, apparatus, device, or storage medium provided in this application, this application determines a target document set based on the business indicator value corresponding to each document in a preset document set and performs a first sorting on the documents in the preset document set to obtain a first sorting result; the target document set includes at least two target documents with the same business indicator value in the preset document set; based on the first sorting result, an initial document number corresponding to each document in the preset document set is determined; based on the keyword sequence corresponding to each target document in the target document set, each target document in the target document set is sorted a second time to obtain a second sorting result; the keyword sequence corresponding to each target document is determined based on the keywords corresponding to each target document; based on the second sorting result, a rearranged document number of each target document in the target document set is determined; and an inverted index database is constructed based on the initial document number corresponding to the non-target documents in the preset document set and the rearranged document number corresponding to each target document in the target document set. For target document sets with the same business indicator value, this application re-sorts the initial document numbers of the target documents in the target document set based on the keywords corresponding to each target document, thereby reducing the distance between document numbers of similar documents and improving the accuracy and efficiency of document queries.

[0300] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0301] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0302] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer storage medium, such as a read-only memory, a disk, or an optical disk.

[0303] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A database construction method characterized by comprising: The method includes: Based on the business indicator value corresponding to each document in the preset document set, the documents in the preset document set are sorted for the first time to obtain a first sorting result; documents with the same business indicator value in the preset document set are taken as target documents to obtain a target document set; the target document set includes at least two target documents with the same business indicator value in the preset document set; Based on the first sorting result, determine the initial document number corresponding to each document in the preset document set; Obtain the keywords corresponding to each target document in the target document set; determine the keywords of each target document in the target document set to obtain a keyword set; Determine the frequency of occurrence of each keyword in the keyword set; sort each keyword in the keyword set according to its frequency of occurrence from smallest to largest to obtain a keyword ranking result; based on the keyword ranking result, determine the first number of keywords with the highest ranking as target keywords; Based on the sorting results corresponding to the first number of target keywords, the position number corresponding to each target keyword is determined; the position number represents the position of the assigned value of the target keyword in the keyword sequence; The target keywords are assigned a first value; a keyword sequence for each target document is generated based on the keywords corresponding to each target document, the position number corresponding to each target keyword, and the first value. Based on the keyword sequence corresponding to each target document in the target document set, the target documents in the target document set are sorted a second time to obtain a second sorting result; the keyword sequence corresponding to each target document is determined based on the keywords corresponding to each target document. Based on the second sorting result, determine the rearranged document number of each target document in the target document set; An inverted index database is constructed based on the initial document numbers corresponding to non-target documents in the preset document set and the rearranged document numbers corresponding to each target document in the target document set.

2. The method according to claim 1, characterized in that, The method further includes: determining the number of digits in the keyword sequence based on the first quantity; generating the keyword sequence for each target document based on the keywords corresponding to each target document, the position number corresponding to each target keyword, and the first value, including: The keywords corresponding to any target document are used as preset keywords to obtain a preset keyword set; The number of target keywords in the preset keyword set is determined to obtain the second number; If the second quantity is less than the first quantity, the blank position in the target keyword sequence is determined according to the position number corresponding to each target keyword in the preset keyword set and the number of digits in the keyword sequence; the target keyword sequence is the keyword sequence corresponding to any target document; the blank position is the position where no target keyword is assigned, determined based on the number of digits in the target keyword sequence. The blank positions in the target keyword sequence are assigned a second value; The target keyword sequence is generated based on the position number corresponding to the target keyword in the preset keyword set, the first value, and the second value.

3. The method according to claim 1, characterized in that, The step of performing a second sorting of the target documents in the target document set based on the keyword sequence corresponding to each target document in the target document set, to obtain the second sorting result, includes: Obtain the first and second values ​​of the keyword sequences corresponding to any two target documents in the target document set, and use them as the current values ​​corresponding to the two target documents. Compare the current values ​​corresponding to any two target documents; the two target documents include the first target document and the second target document. If the current value of the first target document is greater than the current value of the second target document, it is determined that the first target document is ranked before the second target document.

4. The method according to claim 3, characterized in that, The step of determining the rearranged document number of each target document in the target document set based on the second sorting result includes: Obtain the initial document number corresponding to each target document in the target document set to obtain the initial target document number set; Based on the second sorting result, the numbers in the initial target document number set are redistributed to each target document in the target document set to obtain the rearranged document number of each target document in the target document set; wherein, the rearranged document number of the first target document is less than the rearranged document number of the second target document.

5. The method according to claim 1, characterized in that, The process of determining the keywords for each target document in the target document set to obtain a keyword set includes: Extract keywords from the text corresponding to each target document to obtain the keyword set; Determining the frequency of occurrence of each keyword in the keyword set includes: Determine the number of target documents corresponding to each keyword; The number of target documents corresponding to each keyword is taken as the frequency of occurrence of each keyword.

6. The method according to claim 1, characterized in that, The process of determining the keywords for each target document in the target document set to obtain a keyword set includes: Obtain the text, anchor points, and breadcrumbs corresponding to each target document in the target document set; Extract keywords from the text corresponding to each target document to obtain the first keyword for each target document; Extract the keywords from the anchor points corresponding to each target document to obtain the second keyword for each target document; Extract keywords from the breadcrumbs corresponding to each target document to obtain the third keyword corresponding to each target document; The first keyword, the second keyword, and the third keyword corresponding to each target document are determined as the keywords of each target document; The set of keywords composed of the keywords of each target document in the target document set is determined as the keyword set.

7. The method according to claim 6, characterized in that, Determining the frequency of occurrence of each keyword in the keyword set includes: Based on the target document corresponding to each keyword, determine the number of texts, anchor points, and breadcrumbs corresponding to each keyword; The sum of the number of texts, anchor points, and breadcrumbs corresponding to each keyword is determined as the frequency of occurrence for each keyword.

8. The method according to claim 1, characterized in that, The method further includes: Based on the document query information, a set of candidate document IDs matching the document query information is searched from the inverted index database; the candidate document IDs in the candidate document ID set are sorted in order according to the size of their respective document IDs. From the candidate document number set, a preset number of candidate document numbers are selected to obtain the document number set to be queried; the document number set to be queried includes the preset number of candidate document numbers that are sorted at the beginning or end of the candidate document number set; The set of documents to be queried is determined based on the set of document IDs.

9. A database construction apparatus, characterized in that, The device includes: The first sorting module is used to sort the documents in the preset document set for the first time according to the business indicator value corresponding to each document in the preset document set, and obtain a first sorting result; the documents with the same business indicator value in the preset document set are used as target documents to obtain a target document set; the target document set includes at least two target documents with the same business indicator value in the preset document set. The initial document number determination module is used to determine the initial document number corresponding to each document in the preset document set based on the first sorting result. The keyword acquisition module is used to acquire the keywords corresponding to each target document in the target document set; A keyword sequence generation module is used to generate a keyword sequence for each target document based on the keywords corresponding to each target document in the target document set. The keyword sequence generation module includes: a keyword set determination submodule, used to determine the keywords of each target document in the target document set to obtain a keyword set; an occurrence frequency determination submodule, used to determine the occurrence frequency of each keyword in the keyword set; a target keyword determination submodule, used to determine target keywords based on the occurrence frequency of each keyword; sorting each keyword in the keyword set according to its occurrence frequency in ascending order to obtain a keyword ranking result; and determining the first few keywords ranked highest as target keywords based on the keyword ranking result; and a keyword sequence generation submodule, used to generate a keyword sequence for each target document based on the keywords corresponding to each target document and the target keywords. A number digit determination module is used to determine the number of digits in the keyword sequence based on the first quantity; The position number determination module is used to determine the position number corresponding to each target keyword based on the sorting results corresponding to the first number of target keywords; the position number represents the position of the assigned value of the target keyword in the keyword sequence; The keyword sequence generation submodule includes: a first assignment unit, used to assign a first value to the target keyword; and a keyword sequence generation unit, used to generate a keyword sequence for each target document based on the keyword corresponding to each target document, the position number corresponding to each target keyword, and the first value. The second sorting module is used to sort each target document in the target document set a second time according to the keyword sequence corresponding to each target document in the target document set, and obtain a second sorting result; the keyword sequence corresponding to each target document is determined based on the keywords corresponding to each target document. The rearranged document number determination module is used to determine the rearranged document number of each target document in the target document set based on the second sorting result; The database construction module is used to construct an inverted index database based on the initial document numbers corresponding to non-target documents in the preset document set and the rearranged document numbers corresponding to each target document in the target document set.

10. The apparatus according to claim 9, characterized in that, The keyword sequence generation unit includes: The preset keyword set determination sub-unit is used to take the keywords corresponding to any target document as preset keywords to obtain the preset keyword set; The second quantity determination subunit is used to determine the quantity of target keywords in the preset keyword set to obtain the second quantity; The blank position determination subunit is used to determine the blank position in the target keyword sequence based on the position number corresponding to each target keyword in the preset keyword set and the number of digits in the keyword sequence if the second quantity is less than the first quantity; the target keyword sequence is the keyword sequence corresponding to any target document; the blank position is the position where no target keyword is assigned, determined based on the number of digits in the target keyword sequence. The second assignment subunit is used to assign a second value to the blank positions in the target keyword sequence; The target keyword sequence generation subunit is used to generate the target keyword sequence based on the position number corresponding to the target keyword in the preset keyword set, the first value, and the second value.

11. The apparatus according to claim 9, characterized in that, The second sorting module includes: The current value determination submodule is used to obtain the first and second values ​​of the keyword sequences corresponding to any two target documents in the target document set, and use them as the current values ​​corresponding to the two target documents. The comparison submodule is used to compare the current values ​​corresponding to any two target documents; the two target documents include a first target document and a second target document. The sorting submodule is used to determine that the sorting of the first target document is before that of the second target document if the current value of the first target document is greater than the current value of the second target document.

12. The apparatus according to claim 11, characterized in that, The document numbering determination module includes: The initial target document number set acquisition submodule is used to acquire the initial document number corresponding to each target document in the target document set, thereby obtaining the initial target document number set; The numbering allocation submodule is used to reallocate the numbers in the initial target document numbering set to each target document in the target document set according to the second sorting result, so as to obtain the rearranged document number of each target document in the target document set; wherein, the rearranged document number of the first target document is less than the rearranged document number of the second target document.

13. The apparatus according to claim 9, characterized in that, The keyword set determination submodule includes: A keyword set determination unit is used to extract keywords from the text corresponding to each target document to obtain the keyword set; The frequency determination submodule includes: a target document quantity determination unit, used to determine the number of target documents corresponding to each keyword; and a first frequency determination unit, used to take the number of target documents corresponding to each keyword as the frequency of occurrence of each keyword.

14. The apparatus according to claim 9, characterized in that, The keyword set determination submodule includes: The text acquisition unit is used to acquire the text, anchor points, and breadcrumbs corresponding to each target document in the target document set; The first keyword extraction unit is used to extract keywords from the text corresponding to each target document to obtain the first keyword corresponding to each target document. The first keyword extraction unit is used to extract keywords from the anchor points corresponding to each target document to obtain the second keyword corresponding to each target document. The third keyword extraction unit is used to extract keywords from the breadcrumbs corresponding to each target document to obtain the third keyword corresponding to each target document. The keyword determination unit is used to determine the first keyword, the second keyword, and the third keyword corresponding to each target document as the keywords of each target document; The keyword set determination unit is used to determine the set of keywords of each target document in the target document set as the keyword set.

15. The apparatus according to claim 14, characterized in that, The frequency determination submodule includes: The quantity determination unit is used to determine the number of texts, anchor points, and breadcrumbs corresponding to each keyword based on the target document corresponding to each keyword. The second frequency determination unit is used to determine the frequency of occurrence of each keyword by summing the number of texts, anchor points, and breadcrumbs corresponding to each keyword.

16. The apparatus according to claim 9, characterized in that, The device further includes: The candidate document ID set lookup module is used to search for a set of candidate document IDs that match the document query information from the inverted index database based on the document query information; the candidate document IDs in the candidate document ID set are sorted in order according to the size of their respective document IDs; The document ID set determination module is used to filter out a preset number of candidate document IDs from the candidate document ID set to obtain the document ID set to be queried; the document ID set to be queried includes the preset number of candidate document IDs that are sorted at the beginning or end of the candidate document ID set; The document set determination module is used to determine the document set to be queried based on the document number set to be queried.

17. A computer storage medium, characterized in that, The computer storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the database construction method as described in any one of claims 1-8.

Images