Data query methods, devices, equipment, media, and products

Abstract

This application discloses a data query method, apparatus, device, medium, and product, applicable to the field of data processing technology. The method includes: obtaining multiple search terms from a query request, and obtaining the inverted index of each search term; dividing the multiple search terms into a first group and a second group based on the length of the inverted index of each search term; counting the hits of documents in a target document set based on documents related to the inverted index of the first search term; filtering out a first recall document and a document set to be verified from the target document set based on the hit count results; performing document recall verification on the document set to be verified using documents related to the inverted index of the second search term, to filter out a second recall document that meets the document recall conditions from the document set to be verified; and constructing a query result using the first and second recall documents. Using this application embodiment helps improve the efficiency of data querying.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to data query methods, apparatus, equipment, media and products. Background Technology

[0002] Currently, existing data query methods typically involve maintaining a min-heap of size equal to the number of search terms, and then adjusting the elements in the min-heap by comparing and swapping document identifiers to find documents that meet the recall criteria.

[0003] However, the inventors discovered in practice that current data query methods involve a large number of comparisons and exchanges of document identifiers. When the number of documents is large, significant computation time is required, resulting in low data query efficiency. Therefore, improving data query efficiency is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides a data query method, apparatus, device, medium, and product that can group documents based on the length of the inverted index of the search terms. This allows for a first-group inverted index to count the hits of all documents in the target document set, followed by a second-group inverted index to retrieve documents with recall potential (i.e., the set of documents to be verified). This eliminates the need to process all inverted indexes of the search terms (especially longer ones) to determine the retrieved documents, thereby reducing data processing volume and improving data query efficiency.

[0005] On one hand, embodiments of this application disclose a data query method, the method including: Retrieve multiple search terms contained in the search text from the query request, and obtain the inverted index of each search term; the inverted index of each search term is obtained by sorting the document identifiers of the documents containing each search term. Based on the length of the inverted chain of each search term, multiple search terms are divided into a first group and a second group; the length of the inverted chain of the first search term in the first group is less than or equal to the length of the inverted chain of the second search term in the second group; Based on the inverted chain related documents of the first search term, the documents in the target document set are counted for hits. Based on the hit count results, the first recall documents that meet the document recall conditions and the document verification set that meet the document verification conditions are selected from the target document set. By using documents related to the inverted chain of the second search term, document recall verification is performed on the set of documents to be verified in the target document set, so as to filter the second recall documents that meet the document recall conditions from the set of documents to be verified. The query results for the retrieved text are constructed using the first and second recall documents.

[0006] On one hand, embodiments of this application disclose a data query device, the device comprising: The request parsing module is used to obtain multiple search terms contained in the search text from the query request, and to obtain the inverted index of each search term; the inverted index of each search term is obtained by sorting the document identifiers of the documents containing each search term. The grouping module is used to divide multiple search terms into a first group and a second group based on the length of the inverted chain of each search term; the length of the inverted chain of the first search term in the first group is less than or equal to the length of the inverted chain of the second search term in the second group; The first group counting module is used to count the hits of documents in the target document set based on the inverted chain related to the first search term, and to filter out the first recall documents that meet the document recall conditions and the set of documents to be verified that meet the document verification conditions based on the hit count results. The recall verification module is used to perform document recall verification on the set of documents to be verified in the target document set by using documents related to the inverted chain of the second search term, so as to filter the second recall documents that meet the document recall conditions from the set of documents to be verified. The result determination module is used to construct query results for the retrieved text using the first and second recall documents.

[0007] One embodiment of this application provides a computer device, including a memory and a processor; the memory is connected to the processor, the memory is used to store a computer program, and the processor is used to invoke the computer program so that the computer device performs the method provided in any of the above embodiments.

[0008] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having a processor performs the method provided in any of the above aspects.

[0009] One aspect of this application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The computer program is adapted to be read and executed by a processor so that a computer device having a processor performs the method provided in any of the above aspects.

[0010] Using the embodiments of this application, multiple search terms contained in the search text can be obtained from the query request, and the inverted index of each search term can be obtained. Then, based on the length of the inverted index of each search term, the multiple search terms can be divided into a first group and a second group, wherein the length of the inverted index of the first search term in the first group is less than or equal to the length of the inverted index of the second search term in the second group. Furthermore, based on the documents related to the inverted index of the first search term, the documents in the target document set can be counted for hits. Based on the hit count results, the first recall documents that meet the document recall conditions and the set of documents to be verified that meet the document verification conditions can be selected from the target document set. Then, the documents related to the inverted index of the second search term can be used to perform document recall verification on the set of documents to be verified in the target document set, so as to select the second recall documents that meet the document recall conditions from the set of documents to be verified. Finally, the query results for the search text can be constructed using the first recall documents and the second recall documents. Based on this, the inverted chains of the search terms can be grouped. The first group can be used to count the hits of all documents in the target document set through the inverted chains of the first group. Then, the second group can be used to recall and verify the documents with recall potential (i.e., the set of documents to be verified). This eliminates the need to process all inverted chains of the search terms (especially some long inverted chains) to determine the documents to be recalled, thereby reducing the amount of data processing and improving the efficiency of data query. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application; Figure 2 This is a schematic diagram of a product purchasing method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a data query method provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the effect of hit counting provided in an embodiment of this application; Figure 5 This is a flowchart illustrating a document retrieval process provided in an embodiment of this application; Figure 6 This is a flowchart illustrating a data query method provided in an embodiment of this application; Figure 7 This is a flowchart illustrating a window sliding process provided in an embodiment of this application; Figure 8 This is a flowchart illustrating a document retrieval process provided in an embodiment of this application; Figure 9 This is a schematic diagram of a document retrieval process provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a data query device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0014] The following describes some technical terms used in the embodiments of this application: An inverted list (or postings list) is an ordered list of all document identifiers containing a given search term. This inverted list can be a core data structure for data retrieval, storing information about the occurrence of a search term within a document collection. Within a search term's inverted list, the documents containing that search term (i.e., all documents containing the search term) can be arranged in ascending order of their DocId (document identifier).

[0015] Document (Document): In a data retrieval scenario, a document that can be searched. In this application, a document can be published content, users, products, web pages, or other data; no limitation is made here.

[0016] DocId (Document Identifier): This is a unique identifier assigned to each document, i.e., a document identifier. DocIds can be used to arrange documents in ascending order in the index to facilitate sequential indexing during document retrieval.

[0017] Published content: This refers to content information pre-published by users or AI agents on a content platform. This can include text, images, audio, and video, and can be displayed in formats such as notes, articles, and video files. The specific content information included in published content can be adjusted accordingly for different scenarios, and the content information in the published content can be determined by the publisher.

[0018] Term (search term): refers to the smallest indexable unit obtained after segmenting and processing the search text. The inverted chain corresponding to each term records the document information (specifically, document identifiers) of the documents containing that search term.

[0019] DF (Document Frequency): Term frequency refers to the number of documents containing a given search term (i.e., documents containing that search term). It is used to evaluate term sparsity and search optimization strategies. It's important to understand that when calculating term frequency, only the number of documents containing a search term needs to be counted, not the number of times the search term appears in each document. For example, in some cases, if a search term appears only in a single document, and that document contains multiple occurrences of the search term, the term frequency is still calculated as 1, meaning the number of documents containing that search term is considered 1.

[0020] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application. For example... Figure 1 As shown, the data processing system may include terminal devices (such as device 11a, device 12a, and device 13a) and server 100a. It is understood that... Figure 1 The number of terminal devices and servers shown is merely illustrative; any number of terminal devices and servers can be used as needed. Terminal devices (such as devices 11a, 12a, and 13a) can communicate with servers via a network (i.e., through a medium that provides a communication link via wired, wireless communication links, or fiber optic cables, etc.) to transmit data.

[0021] It is understood that a client can run on the terminal device (such as device 12a), which can be a program that provides local services to users (also known as business objects or operation objects). Server 100a can be the server corresponding to the client, and a program that provides resources, service data, and other services to the client can run on server 100a. It is understood that the client running on the terminal device can also be called an application client, business client, etc., and a business account can be logged in on the client. The account currently logged in on the client can be called the current account. For example, the client running on the terminal device can be a client for social networking or communication, and then the business object (such as user U1) can perform data queries on the client, such as searching for published content on a content platform. In the embodiments of this application, the business object (such as user U1) can perform query operations on the terminal device, and then the terminal device can send a query request to the server, so that when the server receives the query request, it can perform document retrieval based on the search text in the query request, determine the query results from the retrieved documents, and return the query results to the user's terminal device.

[0022] It is understood that terminal devices (such as device 11a) may include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, smart speakers, etc., without limitation. The server 100a may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, without limitation.

[0023] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating a product purchasing method provided in an embodiment of this application. For example... Figure 2 As shown, device 11a can be a user's (e.g., the user corresponding to the target account) terminal device. The user can perform a query operation on device 11a, and then device 11a sends a query request to server 100a (step S21a). Furthermore, server 100a can process the query request to obtain the query result and return the query result to device 11a (step S22a). Specifically, server 100a, upon receiving the query request (e.g., the user's terminal device), can perform a query operation on device 11a. Figure 2 When (as shown in 201a), the search text (such as...) can be determined from the query request. Figure 2 As shown in 211a), the search text can then be segmented to obtain multiple search terms included in the search text, such as... Figure 2The search terms shown in 202a are Ta, Tb, ..., Tn. Furthermore, the inverted index corresponding to each search term can be obtained, for example, ... Figure 2 As shown in 203a, the inverted chain La corresponding to the search term Ta, the inverted chain Lb corresponding to the search term Tb, ..., and the inverted chain Ln corresponding to the search term Tn can be obtained.

[0024] Furthermore, multiple search terms can be grouped according to the length of their inverted index (i.e., the term frequency), resulting in Group 1 (e.g., ...). Figure 2 (as shown in 204a) and group 2 (as shown in 204a) Figure 2 (As shown in 205a). Further, the hit count for each document in set 206a can be performed based on the inverted index of the search terms in group 1 to obtain the hit count for each document in set 206a. Set 206a can be a set of documents used for recall calculation; for example, set 206a may include documents D1, D2, D3, D4, and D5. Therefore, the hit count for document D1 (1), document D2 (2), document D3 (3), document D4 (4), and document D5 (5) can be determined.

[0025] Furthermore, documents that meet the document recall criteria can be filtered from set 206a (such as...). Figure 2 As shown in 207a), and a set of documents that meet the document validation conditions (such as...). Figure 2 (As shown in 208a). Document recall conditions are the conditions that must be met for a document to be recalled. For example, the document recall condition could be that the number of hits for a document is greater than or equal to a recall threshold. Document verification conditions are the conditions that must be met for document recall verification. Documents in the document set that meet the document verification conditions have the potential for recall, and further document recall verification can be performed on these documents to determine whether they should be recalled. Furthermore, document recall verification can be performed on the documents in the document set that meet the document verification conditions, and documents that meet the document recall conditions can be selected from the document set that meets the document verification conditions (e.g., ...). Figure 2 (As shown in 209a). Furthermore, it can be based on, as... Figure 2 Documents that meet the document recall criteria, as shown in 207a, and such as Figure 2 Documents that meet the document recall criteria, as shown in 209a, are used to determine the query results for the query request (such as...). Figure 2 As shown in 210a), documents that meet the document recall criteria can be directly returned to device 11a, or documents that meet the document recall criteria can be further filtered or sorted before being returned to device 11a. No limitation is made here.

[0026] The embodiments of this application employ a hybrid retrieval architecture based on data distribution characteristics. Unlike traditional algorithms that use a single processing mode for all inverted chains, this embodiment splits the inverted chains into a "sparse flow" (i.e., the first group) and a "dense flow" (i.e., the second group) based on document frequency (or the chain length). This achieves adaptive matching between the algorithm strategy and data features (i.e., the length of the inverted chains). For sparse data (i.e., shorter inverted chains), it utilizes the linear processing power of the CPU, while for dense data, it leverages the skip-lookup capability of the index, thus breaking the O(NlogN) performance bottleneck and reducing the overall complexity to linear O(N). Furthermore, it provides adaptive optimization strategies for different inverted chains, achieving highly compatible query performance improvements and adapting to search needs in various recall scenarios.

[0027] It should be noted that all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of relevant user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, this application may display prompts before and during the collection of user data to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt; otherwise (i.e., without receiving confirmation from the user), the steps for collecting user data are terminated, meaning that user data is not collected.

[0028] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0029] Based on the above description, this application proposes a data query method. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a flowchart illustrating a data query method provided in an embodiment of this application. The method can be executed by a computer device, such as a server 100a. The data query method may include the following steps S101-S105.

[0030] S101. Obtain multiple search terms contained in the search text from the query request, and obtain the inverted chain of each search term; the inverted chain of each search term is obtained by sorting the document identifiers of the documents containing each search term.

[0031] The query request can be a request used to perform a data query. The query request can carry search text, which can be text used for data querying. The search text can contain multiple search terms, which can be words obtained by text segmentation of the search text. Text segmentation refers to dividing a piece of text into smaller semantic units (such as words, characters, etc.). For example, if the search text is "common methods for data querying," then text segmentation of the search text can yield the search terms "data," "query," "of," "common," and "method."

[0032] As explained above regarding inverted chains, each search term can have a corresponding inverted chain. The inverted chain for each search term is obtained by sorting the document identifiers of the documents containing that search term. The document containing the search term refers to any document that contains that search term. For example, if the search term "Ta" appears in document D1, then document D1 can be the document containing the search term "Ta". Sorting the document identifiers of the documents containing each search term can be done in ascending order or descending order. In this application, we will primarily use ascending order as an example. For instance, if the documents containing the search term "Ta" include documents D1, D4, D6, D7, D9, and D10, where D1, D4, and D10 are document identifiers, the inverted chain corresponding to the search term "Ta" could be: D1-D4-D6-D7-D9-D10. As you can see, this inverted chain can be sorted in ascending order of document identifiers.

[0033] S102. Based on the length of the inverted chain of each search term, divide multiple search terms into a first group and a second group; the length of the inverted chain of the first search term in the first group is less than or equal to the length of the inverted chain of the second search term in the second group.

[0034] In this context, the inverted index chain length refers to the number of document identifiers within the inverted index chain. It's understood that the chain length of an inverted index for a search term is the same as its term frequency (see the description above for an explanation of term frequency). Therefore, based on the chain length of each search term's inverted index, multiple search terms are divided into a first group and a second group, which is equivalent to dividing multiple search terms into a first group and a second group based on their term frequency. For example, if the inverted index is D1-D4-D6-D7-D9-D10, then the chain length is 6, meaning the term frequency is 6.

[0035] The first group refers to the group of search terms used for full document counting, also known as the traversal group. The second group refers to the group of search terms used for recall verification, also known as the verification group. In the first group, the inverted index length of the first search term is less than or equal to that of the second search term in the second group. It can be understood that the first group contains K search terms with shorter inverted index lengths (or lower term frequencies), whose inverted indexes are sparse and short, suitable for full document scanning. The second group contains inverted indexes of multiple search terms excluding those in the first group, where the inverted indexes of these terms are longer (or higher term frequencies), whose inverted indexes are dense and long, suitable for on-demand seek searches (Seek), i.e., suitable for recall verification.

[0036] In one possible implementation, dividing multiple search terms into a first group and a second group based on the chain length of the inverted chain of each search term may include the following steps: 1. Obtaining a recall threshold associated with the query request, and then determining the number of group terms based on the recall threshold and the number of terms in the multiple search terms; the number of group terms is K, where K is a positive integer; 2. Sort the multiple search terms according to the chain length of the inverted chain of each search term; 3. Among the multiple search terms, determining the K search terms with the smallest inverted chain length as the first group, and determining the search terms other than those in the first group as the second group.

[0037] The recall threshold associated with a query request can refer to the minimum number of times a document needs to be matched to retrieve it, i.e., the minimum matching threshold. The number of times a document is matched can refer to the number of times multiple search terms appear in the document; a match here means that the search terms appear in the document. For example, if the recall threshold is 3, it means that to retrieve a document, the number of matches for that document must be greater than or equal to 3. In other words, at least 3 search terms contained in the search text need to match the document to retrieve it.

[0038] The number of grouped terms can be used to determine the number of search terms in the first group when grouping search terms. In other words, the number of search terms in the first group is the number of terms in that group. The number of grouped terms is K, where K is a positive integer.

[0039] Optionally, the calculation method for the number of grouped words can be found in the following formula (1): K=TotalTerms-minShouldMatch+1 formula (1) Among them, TotalTerms can be the number of terms for multiple search terms, and minShouldMatch can be the recall threshold.

[0040] For example, when the number of terms in multiple search terms (i.e., TotalTerms) is 5, and the recall threshold (minShouldMatch) is 3, that is, among the multiple search terms included in the retrieved text, at least 3 search terms need to match the document to recall the document. Then, K = 5 - 3 + 1 = 3, that is, the number of search terms in the first group is 3. Based on this, after dividing into the first group and the second group, after determining the hit count of each document in the first group in full volume, documents that cannot meet the recall condition even if each search term in the second group matches the document can be quickly screened out, thus avoiding waste of computing resources and improving the efficiency of document recall.

[0041] 进一步的,可以按照每个检索词的倒排链的链长的大小,对多个检索词进行排序,然后,在多个检索词中,将倒排链的链长最小化的K个检索词确定为第一分组,并将多个检索词中除第一分组之外的检索词确定为第二分组。例如,当前查询请求的检索文本中包含5个检索词(TermA,TermB,TermC,TermD,TermE),且系统设定的召回阈值(即minShouldMatch)为3。即文档必须至少包括3个检索词才算召回。那么,根据公式1计算出第一分组中的检索词数量为K=5

[0041] 3+1=3,在对多个检索词排序后顺序为A<B<C<D<E,那么,第一分组中可以包括:TermA,TermB,TermC,第二分组中可以包括:包含TermD,TermE。

[0042] S103. Perform hit counting on the documents in the target document set based on the posting list related to the first search term, and screen out the first recalled documents that meet the document recall condition and the set of documents to be verified that meet the document verification condition from the target document set based on the hit counting result.

[0043] Among them, the target document set can be a document set used for batch recall calculation. Among the documents in the target document set, the documents can be N consecutive documents with document identifiers. For example, the target document set can be a set of documents with document identifiers from 1 to 1000.

[0044] Optionally, the target document set can be a collection of all or part of the documents in the document library. The document library can be the full set of documents in the content platform requested by the query request. Optionally, the target document set can be determined from the document library according to a preset window size, and the target document set can be updated by sliding the window to include all documents in the document library as documents in the target document set for recall calculation. This allows for batch recall calculation of documents in the document library, avoiding excessive document volume during a single recall calculation, helping to reduce the time complexity of recall calculation and improve the flexibility of recall calculation.

[0045] Hit counting refers to the process of statistically analyzing the number of times multiple search terms match documents. The hit count result can include the number of hits for each document in the target document set. Specifically, counting the hits of documents in the target document set based on the inverted index related to the first search term can be done by counting the number of times multiple search terms match each document in the target document set. In other words, it can be done by counting the number of times documents in the target document set appear in the inverted index related to the first search term, and then determining the count as the hit count.

[0046] In one possible implementation, the target document set includes N documents, where N is a positive integer. Then, counting the hits of documents in the target document set based on the inverted index of the first search term can include the following steps: 1. Obtaining a document counter associated with the target document set; the document counter includes N counting positions, each counting position corresponding to a document in the target document set; 2. If the inverted index of each search term in the first group includes the document identifier of the target document in the target document set, then the cumulative occurrence count of the document identifier of the target document in the inverted index of each search term in the first group is determined as the hit count corresponding to the target document; 3. Determining the counting position corresponding to the target document in the document counter, and updating the count value at the counting position corresponding to the target document to the hit count corresponding to the target document, so as to determine the hit count result corresponding to the target document set based on the count value at each counting position in the document counter.

[0047] The document counter, also known as a CounterMap, is used to record the number of hits for documents in the target document set. It consists of N counting positions, each corresponding to a document in the target document set. The number of counting positions is the same as the number of documents in the target document set, and each counting position is associated with a document's document identifier. Each counting position records a hit count, with the count value representing the number of hits for a given document. For example, if the target document set contains 1000 documents with document identifiers D1-D1000, then the document counter associated with the target document set would have 1000 counting positions, each corresponding to a document. For instance, document D1 would correspond to the first counting position, document D2 to the second, and so on.

[0048] Optionally, the document counter can be represented in various ways, such as an array, a hash table, or a bitmap; no further limitations are specified here. When using an array as the document counter, the array index can be mapped to the document identifier, and each element in the array serves as a counting position to record the number of times the corresponding document has been hit. A hash table is a data structure that maps keys (such as document identifiers) to values ​​(such as hit counts). It uses a hash function to convert the key (such as the document identifier) ​​into an array index, recording the hit count at the element position (i.e., the counting position) corresponding to the converted array index. A bitmap is a data structure that uses bits to represent the existence or state of data. Each element (such as the document identifier) ​​is mapped to a bit (i.e., the counting position), and the hit count is recorded through the value at the bit.

[0049] The target document can be any document in the target document set. In this application, each document in the target document set can be used as the target document in sequence according to its document identifier, and the hit count can be performed on the target documents to obtain the hit count for each document in the target document set, thereby obtaining the hit count result.

[0050] The cumulative occurrence count can refer to the cumulative number of times the document identifier of the target document appears in the inverted index of each search term in the first group. For example, if the search terms in the first group include TermA, TermB, and TermC, then the inverted indexes of TermA, TermB, and TermC can be traversed. If document D1 appears in the inverted indexes of TermA and TermB, then the cumulative occurrence count of document D1 in the inverted index of each search term in the first query term is 2. Therefore, the hit count of document D1 can be determined as 2, and the count value at the corresponding counting position (such as the first counting position) of document D1 in the document counter can be updated to 2.

[0051] It is understandable that when the document counter is initialized, the count value at each counting position in the document counter is 0. Therefore, when the cumulative occurrence count of the target document in the inverted chain of each search term in the first group is 0, the count value at the counting position corresponding to the target document in the document counter does not need to be updated.

[0052] Optionally, each counting position in the counter is associated with a corresponding position index. When updating the count value in the document counter, the document identifier of the document can be calculated based on the address calculation logic to obtain the position index corresponding to the document, thereby updating the count value at the determined position index. The address calculation logic can be the logic used to determine the position index corresponding to the document's counting position. For example, the address calculation logic could be index = DocId - BasedID, where index can be the position index, DocId is the document identifier, and BasedID can be the document identifier of the first document in the target document set.

[0053] For example, see Figure 4 , Figure 4 This is a schematic diagram illustrating the effect of a hit count provided in an embodiment of this application. For example... Figure 4 As shown, counter 401a (i.e., document counter) can be obtained. Taking a target document set containing 10 documents as an example, the document counter can include 10 counting positions. Each document can have a corresponding counting position, and the mapping relationship between documents and counting positions can be calculated using address calculation logic. For example, multiple search terms include: TermA, TermB, and TermC. By traversing the inverted chains corresponding to each term's hit count, it can be determined that TermA's inverted chain includes D5, TermB's inverted chain includes D5, and therefore document D5 has a hit count of 2. This can be calculated based on address calculation logic (such as...). Figure 4As shown in 402a, if the counting position corresponding to document D5 is determined to be the counting position with position index 5, then the count value at the counting position with position index 5 can be updated to 2. Similarly, if D8 is included in the inverted chain of TermC, then the hit count of document D8 is 1. Based on the address calculation logic, the counting position corresponding to document D8 can be determined to be the counting position with position index 8, and then the count value at the counting position with position index 8 can be updated to 1.

[0054] Understandably, after determining the hit count of each document based on the search terms in the first group, documents that can be recalled, documents with recall potential, and documents with no recall potential can be identified from each document. Recalled documents can be those that meet the document recall criteria; documents with recall potential can be those that do not meet the document recall criteria but whose recall potential information meets certain conditions. These documents with recall potential can be subsequently validated using the search terms in the second group to determine whether they can be recalled. This recall potential information can be used to represent the hit count when all search terms in the second group contain the target document; documents with no recall potential can be those that do not meet the document recall criteria and whose recall potential information does not meet certain conditions. These documents with no recall potential can be discarded, and further recall validation is not required.

[0055] Here, the first recalled document can refer to the recalled document, which can be a document that meets the document recall criteria. The document recall criteria here can be that the number of hits of the document is greater than or equal to the recall threshold. The number of first recalled documents can be one or more.

[0056] The document set to be verified can be a collection of documents with recall potential. Further document recall verification can be performed on these documents to determine their recallability. This document set can also be a collection of documents that meet the document verification criteria. These criteria can be that the document's recall potential information is greater than or equal to a recall threshold. The recall potential information can represent the number of hits when all search terms in the second group contain the target document, representing the document's potential for recall.

[0057] Among them, the recall potential information is determined based on the number of hits and the number of search terms in the second group, as shown in the following formula (2): Formula (2) = Potential = M + L Where Potential represents the recall potential information of the target document. M represents the current hit count, which can be the number of times the document was hit when calculating the recall potential information. L refers to the number of inverted chains in the inverted chain of the search terms in the second group that did not participate in the verification of the recalled document (e.g., the number of chains to be verified) when calculating the recall potential information.

[0058] In one possible implementation, the hit count result includes the number of hits for each document in the target document set. Then, selecting the first recalled document that meets the document recall criteria and the set of documents to be verified that meet the document verification criteria from the target document set based on the hit count result may include the following steps: 1. Determine the number of hits for the target document in the target document set from the hit count result; 2. If the number of hits for the target document is greater than or equal to the recall threshold, then determine the target document as the first recalled document that meets the document recall criteria; 3. If the number of hits for the target document is less than the recall threshold, then determine the recall potential information of the target document based on the number of hits for the target document and the number of second search terms. When the recall potential information of the target document is greater than or equal to the recall threshold, the target document is determined as a document in the set of documents to be verified that meet the document verification criteria.

[0059] For details regarding the hit count results, please refer to the relevant descriptions above; they will not be repeated here. Similarly, for details regarding recall potential information, please refer to the relevant descriptions above; they will not be repeated here.

[0060] In one possible implementation, the embodiments of this application may further include: discarding the target document when the recall potential information is less than the recall threshold.

[0061] Understandably, when the recall potential information is greater than or equal to the recall threshold, it means that the document still has the possibility of being recalled, and further verification is performed. When the recall potential information is less than the recall threshold, it means that the document no longer has the possibility of being recalled. That is, even if the target document exists in all the inverted chains in the second group, the number of hits of the target document is still less than the recall threshold (i.e., the document recall condition is not met), so the target document can be directly discarded, which means that document recall verification is not required, thereby reducing the waste of computing resources.

[0062] This section illustrates the document recall process with illustrations. Please refer to [link / reference]. Figure 5 , Figure 5 This is a flowchart illustrating a document retrieval process provided in an embodiment of this application. For example... Figure 5 As shown, the hit count results are determined based on the inverted index of the query terms in the first group (e.g., Figure 5 (as shown in 501a), and then determine the number of times the target document was hit from the hit count results (e.g., Figure 5 As shown in 502a), further, it is determined whether the number of hits is greater than the recall threshold (e.g., Figure 5 (See 503a). If the number of hits exceeds the recall threshold, the target document is identified as a document to be recalled (e.g., ...). Figure 5 As shown in 504a), the retrieved document here is the first recalled document mentioned above. If the number of hits is less than or equal to the recall threshold, the recall potential information is determined based on the number of hits on the target document and the number of search terms in the second group (e.g., Figure 5 As shown in 505a), further, it is determined whether the recall potential information is greater than the recall threshold (e.g., Figure 5 (See 506a). If the recall potential information is greater than the recall threshold, the target document is identified as a document with recall potential (e.g., ...). Figure 5 As shown in 507a), furthermore, document recall verification can be performed on the target document (such as...). Figure 5 (See 508a). Conversely, if the recall potential information is less than or equal to the recall threshold, the target document is determined to be a document with no recall potential (e.g., ...). Figure 5 As shown in 509a), the target document can then be discarded (e.g. Figure 5 (as shown in 510a).

[0063] S104. By using documents related to the inverted chain of the second search term, perform document recall verification on the set of documents to be verified in the target document set, so as to filter the second recall documents that meet the document recall conditions from the set of documents to be verified.

[0064] The document recall verification step, which involves verifying whether documents in the document set to be verified have been recalled, is a process of confirming that documents in the document set have been retrieved. The second recalled document can refer to documents that meet the document recall criteria and are selected from the document set to be verified during the document recall verification. It is understood that both the first and second recalled documents can be recalled documents and can be added to the document recall list to determine the query results.

[0065] S105. Using the first and second recall documents, construct the query results for the retrieved text.

[0066] The query results can be based on the retrieved text. Optionally, the query results can be constructed using the first and second recalled documents. This can be done by directly using the first and second recalled documents (i.e., documents that meet the document recall criteria) as the query results, or by further filtering or sorting the first and second recalled documents to determine the query results. There are no restrictions on this.

[0067] Using the embodiments of this application, multiple search terms contained in the search text can be obtained from the query request, and the inverted index of each search term can be obtained. Then, based on the length of the inverted index of each search term, the multiple search terms can be divided into a first group and a second group, wherein the length of the inverted index of the first search term in the first group is less than or equal to the length of the inverted index of the second search term in the second group. Furthermore, based on the documents related to the inverted index of the first search term, the documents in the target document set can be counted for hits. Based on the hit count results, the first recall documents that meet the document recall conditions and the set of documents to be verified that meet the document verification conditions can be selected from the target document set. Then, the documents related to the inverted index of the second search term can be used to perform document recall verification on the set of documents to be verified in the target document set, so as to select the second recall documents that meet the document recall conditions from the set of documents to be verified. Finally, the query results for the search text can be constructed using the first recall documents and the second recall documents. Based on this, the inverted chains of the search terms can be grouped. The first group can be used to count the hits of all documents in the target document set through the inverted chains of the first group. Then, the second group can be used to recall and verify the documents with recall potential (i.e., the set of documents to be verified). This eliminates the need to process all inverted chains of the search terms (especially some long inverted chains) to determine the documents to be recalled, thereby reducing the amount of data processing and improving the efficiency of data query.

[0068] Based on the above description, this application proposes a data query method. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a flowchart illustrating a data query method provided in an embodiment of this application. The method can be executed by a computer device, such as a server 100a. The data query method may include the following steps S201-S207.

[0069] S201. Obtain multiple search terms contained in the search text from the query request, and obtain the inverted chain of each search term; the inverted chain of each search term is obtained by sorting the document identifiers of the documents containing each search term.

[0070] S202. Based on the length of the inverted chain of each search term, divide multiple search terms into a first group and a second group; the length of the inverted chain of the first search term in the first group is less than or equal to the length of the inverted chain of the second search term in the second group.

[0071] S203. Based on the inverted chain related documents of the first search term, count the hits of the documents in the target document set. Based on the hit count results, select the first recall document that meets the document recall condition and the document verification set that meets the document verification condition from the target document set.

[0072] The steps S201-S203 can be found in the descriptions of steps S101-S103 above, and will not be repeated here.

[0073] In one possible implementation, the documents in the target document set are all or part of the documents in the document library; each document in the document library has a corresponding document identifier; the hit count result is recorded based on the document counter; then, the embodiments of this application may further include the following steps: 1. Sort the document identifiers of each document in the document library according to the size order of the document identifiers to obtain a document identifier sequence; 2. Determine a document identifier window from the document identifier sequence according to a preset window size, and determine the documents corresponding to the document identifiers in the document identifier window as the target document set; the window size of the document identifier window is the same as the number of counting positions of the document counter; 3. After determining the first recall document and the second recall document from the target document set based on the document counter related to the target document set, slide the document identifier window on the document identifier sequence according to a preset sliding step size; 4. Update the target document set based on the document identifiers in the slid document identifier window, and reset the document counter, and determine the first recall document and the second recall document from the updated target document set based on the reset document counter, until all document identifiers in the document identifier sequence are used as document identifiers in the document identifier window.

[0074] The document identifier sequence can be a sequence obtained by sorting the document identifiers of various documents in the document library. The window size can refer to the size of the document identifier window, specifically the maximum number of documents that can be included in the document identifier window. Furthermore, the document identifier window can be determined from the document identifier sequence, and the documents in the document identifier window can be identified as the target document set.

[0075] The preset sliding step size can be the step size that is slid when the window is slid. This preset sliding step size can be the same as the window size. Based on this, different documents can be identified as the target document set each time, avoiding repeated judgments on whether to recall documents, thereby improving the efficiency of document recall calculation.

[0076] Updating the target document set based on the document identifiers in the slid-down document identifier window can be achieved by using the documents corresponding to the document identifiers in the slid-down document identifier window as the target document set.

[0077] The process of resetting the document counter restores the count value at each counting position to its initial value (e.g., 0). Based on the reset document counter, the first and second recalled documents are determined from the updated target document set. Please refer to the relevant descriptions of steps S103-S104 above, which will not be repeated here.

[0078] For example, see Figure 7 , Figure 7 This is a flowchart illustrating a window sliding process provided in an embodiment of this application. Figure 7 As shown, document library 200a can include multiple documents, and these documents can be sorted in ascending order of their document identifiers. First, a window 201a with a size of 10 can be defined on document library 200a, containing documents with identifiers 1-10; these documents can be identified as the target document set. Further, window 201a can be slid, resulting in window 202a, which can contain documents with identifiers 11-20; the target document set can then be updated to include documents with identifiers 11-20. Similarly, window 202a can be slid, resulting in window 203a, which can contain documents with identifiers 21-30; the target document set can then be updated to include documents with identifiers 21-30. This process continues, allowing all documents in the document library to be included in the sliding windows, thus enabling batch recall of documents from the document library.

[0079] Based on this, the embodiments of this application adopt a heapless sorting mechanism that combines a fixed-length sliding window (i.e., a document identifier window) with a counting array (i.e., a document counter). It abandons the traditional approach of maintaining a min-heap for multi-way merging and proposes a counting scheme of "sliding window + direct address mapping". This eliminates the frequent memory swapping and branch prediction failures caused by heap sorting, significantly improves the CPU cache hit rate, and greatly reduces the number of CPU instructions per query, thereby improving the efficiency of data querying.

[0080] S204. Determine the documents to be verified from the set of documents to be verified.

[0081] The document to be verified can be any document in the document set to be verified. It can be understood that documents in the document set to be verified can be sequentially selected according to their document identifiers to perform document recall verification, thereby determining whether each document in the document set to be verified is recalled, and ultimately finding the second recall document in the document set that meets the document recall criteria.

[0082] S205. Determine the verification inverted chain from the inverted chain of the second search term.

[0083] The verification inverted chain can be used for recalling and verifying documents to be verified. In this application, each inverted chain in the inverted chain of the second search term can be identified as a verification inverted chain in ascending order of chain length, until it is determined whether the document to be verified should be discarded or recalled based on the verification inverted chain.

[0084] It is understandable that each time a verification inverted chain is determined from the inverted chain of the second search term, the number of inverted chains to be verified can be updated, that is, the number of inverted chains to be verified is reduced by 1. Therefore, when performing recall verification based on the current verification inverted chain, the number of inverted chains to be verified that needs to be calculated is the number obtained when the current verification inverted chain is determined.

[0085] S206. Based on the verification inverted chain, perform recall verification on the documents to be verified to obtain the recall verification results for the documents to be verified, and update the documents to be verified based on the set of documents to be verified until each document in the set of documents to be verified is used as a document to be verified for recall verification, so as to select the second recall documents that meet the document recall conditions from the set of documents to be verified; wherein, the recall verification results include a first verification result for indicating that the document to be verified meets the document recall conditions, or a second verification result for indicating that the document to be verified meets the document discard conditions.

[0086] Here, recall verification of the document to be verified refers to the process of verifying whether the document to be verified should be recalled. The recall verification result can be the result of whether the document to be verified has been recalled. The recall verification result includes a first verification result indicating that the document to be verified meets the document recall condition, or a second verification result indicating that the document to be verified meets the document discard condition.

[0087] In one possible implementation, recall verification of the document to be verified based on the verification inverted chain to obtain the recall verification result for the document to be verified may include the following steps: 1. Search for the document identifier of the document to be verified in the verification inverted chain; 2. If the document identifier of the document to be verified is found in the verification inverted chain, the number of hits of the document to be verified in the hit count result is accumulated to obtain the update hit count of the document to be verified; 3. If the update hit count of the document to be verified is greater than or equal to the recall threshold, the first verification result used to indicate that the document to be verified meets the document recall condition is taken as the recall verification result of the document to be verified.

[0088] The process of searching for the document identifier of the document to be verified in the verification inverted chain can be done by directly seeking whether the current document identifier (i.e., the verification document identifier) ​​exists, according to the skip list structure of the inverted chain. If it exists, it means that the document identifier of the document to be verified has been found in the verification inverted chain, and the hit count is updated accordingly. Otherwise, it means that the document identifier of the document to be verified has not been found in the verification inverted chain.

[0089] If the document identifier of the document to be verified is found in the verification inverted list, it means that the document to be verified matches the query term corresponding to that verification inverted list. In this case, the hit count for the document to be verified can be updated, for example, by incrementing the hit count by 1 to obtain the updated hit count for the document to be verified. The updated hit count can refer to the updated hit count.

[0090] Furthermore, it can be determined that the number of update hits of the document to be verified is greater than or equal to the recall threshold. If the number of update hits of the document to be verified is greater than or equal to the recall threshold, the first verification result used to indicate that the document to be verified meets the document recall condition will be used as the recall verification result of the document to be verified.

[0091] Conversely, if the number of update hits for the document to be verified is less than the recall threshold, the recall potential information of the document to be verified can be updated to determine whether the document to be verified still has a recall possibility after being recalled and verified through the verification inverted chain.

[0092] In one possible implementation, the present application embodiment may further include the following steps: 1. If the number of update hits of the document to be verified is less than the recall threshold, the recall potential information of the document to be verified is updated based on the number of update hits of the document to be verified and the number of inverted chains to be verified, to obtain the updated recall potential information of the document to be verified; the inverted chain to be verified refers to the inverted chain in the inverted chain of the second search term that is not used as a verification inverted chain; 2. When the updated recall potential information of the document to be verified is less than the recall threshold, the first verification result used to indicate that the document to be verified meets the document discarding condition is used as the recall verification result of the document to be verified, and the document to be verified is discarded; 3. When the updated recall potential information of the document to be verified is greater than or equal to the recall threshold, the verification inverted chain is updated based on the inverted chain to be verified, and the document to be verified is recalled based on the updated verification inverted chain, until the document verification result of the document to be verified is obtained.

[0093] Among them, updating recall potential information can refer to the recall potential information obtained after updating the recall potential information of the document to be verified.

[0094] The updating of the recall potential information for the document to be verified can be determined based on the number of update hits for the document and the number of inverted chains to be verified. It is understood that inverted chains to be verified refer to those inverted chains of the second search term that are not used as verification inverted chains. It is also understood that before the document to be verified is retrieved, all inverted chains of the second query term are inverted chains to be verified; therefore, the recall potential information for the document to be verified is determined based on the number of hits and the number of inverted chains of the second query term.

[0095] Furthermore, based on the updated recall potential information, it can be determined whether to discard the document to be verified, recall the document, or perform recall verification again based on the next inverted index to be verified. Understandably, when the updated recall potential information of the document to be verified is greater than or equal to the recall threshold, the verification inverted index is updated based on the inverted index to be verified, and the document to be verified is recalled and verified based on the updated verification inverted index until the document verification result is obtained.

[0096] Understandably, the number of inverted chains to be verified when determining the updated recall potential information is determined when identifying the verification inverted chains from the inverted chains of the second query term. For example, if the inverted chains of the second query term include L1, L2, and L3, then when L1 is used as the verification inverted chain, the inverted chains to be verified are L2 and L3, so the number of inverted chains to be verified is 2. Further, after performing recall verification based on inverted chain L1, if the number of update hits of the document to be verified is less than the recall threshold, and the updated recall potential information can be determined based on the number of update hits and the number of inverted chains to be verified (i.e., 2), if the updated recall potential information is greater than the recall threshold, then L2 can be used as the verification inverted chain, and the inverted chain to be verified is L3, so the number of inverted chains to be verified is 1. Further, recall verification is performed based on inverted chain L2, and so on, until it is determined that the document to be verified has been discarded or recalled.

[0097] As described above, the second search term can be sorted in ascending order of chain length. Then, when updating the verification inverted chain based on the inverted chain to be verified, the inverted chain corresponding to the next search term of the current inverted chain to be verified can be used as the new verification inverted chain. The document to be verified is then recalled and verified based on the new verification inverted chain, and so on, until the document verification result of the document to be verified is obtained.

[0098] In one possible implementation, the present application embodiment may further include the following steps: 1. If the document identifier of the document to be verified is not found in the verification inverted chain, the recall potential information of the document to be verified is updated based on the number of hits of the document to be verified and the number of inverted chains to be verified, to obtain the updated recall potential information of the document to be verified, wherein the inverted chain to be verified refers to the inverted chain of the second search term that is not used as the verification inverted chain; 2. When the updated recall potential information of the document to be verified is less than the recall threshold, the first verification result used to indicate that the document to be verified meets the document discarding condition is used as the recall verification result of the document to be verified, and the document to be verified is discarded; 3. When the updated recall potential information of the document to be verified is greater than or equal to the recall threshold, the verification inverted chain is updated based on the inverted chain to be verified, and the document to be verified is recalled based on the updated verification inverted chain, until the document verification result of the document to be verified is obtained.

[0099] If the document identifier of the document to be verified is not found in the verification inverted chain, it means that the verification inverted chain does not contain the document identifier of the document to be verified. In this case, the hit count of the document to be verified does not need to be updated. However, the recall potential information of the document to be verified can be recalculated based on the updated hit count of the document to be verified and the number of inverted chains to be verified (i.e., the updated recall potential information is obtained). The determination of the updated recall potential information and the judgment logic for the updated recall potential information can be found in the relevant descriptions above, and will not be repeated here.

[0100] S207. Using the first recall document and the second recall document, construct the query results for the retrieved text.

[0101] The methods for determining the query results can be found in the relevant descriptions above, and will not be repeated here.

[0102] It is understandable that when the target document set is a collection of some documents in the document library, multiple target document sets can be determined based on window sliding operations, and the first and second recall documents in each target document set can be determined. Thus, based on the first and second recall documents in each target document set, the query results for the searched text can be determined.

[0103] This section illustrates the document retrieval process for a specific document using diagrams. Please refer to [link / reference]. Figure 8 , Figure 8 This is a flowchart illustrating a document retrieval process provided in an embodiment of this application. For example... Figure 8 As shown, based on the term frequency of each search term, multiple search terms are divided into a first group and a second group (e.g., ...). Figure 8As shown in 801a), this refers to the grouping based on the length of the inverted chain corresponding to each search term. Then, the document identifier window is initialized, and the target document set (such as...) is determined based on the document identifier window. Figure 8 (As shown in 802a), then, traverse the inverted chain of the search terms in the first group, and record the number of hits in the document counter (e.g., Figure 8 As shown in 803a). Further, the target documents are determined from the target document set (e.g., Figure 8 As shown in 804a), then, it is determined whether the number of hits on the target document is greater than the recall threshold (e.g., ...). Figure 8 As shown in 805a). Where the number of hits on the target document exceeds the recall threshold, then the target document can be recalled (e.g., ...). Figure 8 As shown in 806a), the target document is identified as the document to be recalled. Further, the next document in the target document set can be identified as the target document, and the subsequent steps are repeated. Conversely, if the number of hits on the target document is less than or equal to the recall threshold, recall potential information (such as...) can be determined based on the number of hits and the number of inverted chains to be verified. Figure 8 As shown in 807a), further, it is determined whether the recall potential information is greater than the recall threshold (e.g. Figure 8 (As shown in 808a). If the recall potential information is less than or equal to the recall threshold, the target document can be discarded (e.g., ...). Figure 8 As shown in 809a), further, the target document can be determined from the target document set, that is, the next document in the target document set is determined as the target document, and the subsequent steps are repeated. Conversely, if the recall potential information is greater than the recall threshold, the verification inverted chain can be determined from the second group, and the hit count is updated based on the verification inverted chain (e.g., Figure 8 As shown in 810a), further, it is determined whether the number of hits on the target document is greater than the recall threshold (e.g., Figure 8 As shown in 805a), the target document at this time is the document to be verified mentioned above. Therefore, it is possible to determine whether the target document has met the document recall condition based on the updated hit count, and then repeat the subsequent steps, and so on, until all documents in the target document set have been judged.

[0104] This section illustrates the modules and interaction processes involved in document retrieval, using diagrams. Please refer to [link / reference]. Figure 9 , Figure 9 This is a schematic diagram of a document retrieval process provided in an embodiment of this application. For example... Figure 9 As shown, you can obtain query requests (such as...) Figure 9 As shown in 901a), furthermore, word frequency analysis and grouping units (such as...) can be used to analyze word frequency and grouping units (such as...) Figure 9 As shown in 902a), the multiple search terms included in the search text of the query request are grouped into a first group (e.g., Figure 9 (as shown in 903a) and the second group (as shown in 903a) Figure 9 (As shown in 904a). Furthermore, this can be achieved through a document counter (such as...). Figure 9 As shown in 905a, the number of times each search term in the first group is matched in each document of the target document set is recorded. Then, the target documents are determined from the target document set, and this can be done through a pruning controller (such as...). Figure 9 As shown in 906a, the target document is pruned. This pruning controller determines whether to discard or recall the target document. Specifically, when the pruning controller determines that the number of hits on the target document meets the document recall criteria, the target document can be added to the recall document list as a recalled document (e.g., ...). Figure 9 As shown in 908a). When the recall potential information of the target document is less than the recall threshold, the target document is discarded (e.g., ...). Figure 9 As shown in 907a). When the recall potential information of the target document is greater than or equal to the recall threshold, it is verified by the recall verification unit (such as...). Figure 9 As shown in 909a in the figure, the second group is processed to perform document recall verification on the target document. At this time, the target document is the document to be verified mentioned above. Then, the recall verification unit updates the number of hits and the number of inverted chains to be verified, so that the pruning controller can determine whether to discard or recall the target document based on the updated number of hits and the number of inverted chains to be verified.

[0105] In this embodiment, a multi-stage dynamic pruning and early termination logic is used for document retrieval. During the retrieval verification phase, the theoretical maximum potential value (i.e., retrieval potential information) is calculated in real time by combining the "hit count" and the "remaining unverified chain count" (i.e., the number of inverted chains to be verified). Once it is confirmed that the retrieval potential information cannot meet the retrieval threshold, the verification operation for subsequent long chains is immediately stopped; once the document retrieval conditions are met, the result is returned immediately (i.e., the document is retrieved). Based on this, I / O and decoding overhead can be minimized, and in long-tail query scenarios, a large number of invalid underlying index reads can be avoided, significantly reducing P99 latency.

[0106] This section uses a specific example to illustrate the entire document retrieval process. To more intuitively explain the execution process of this method, assume that the current query contains 5 search terms (TermA, TermB, TermC, TermD, TermE), and the set minimum matching threshold (i.e., the recall threshold, minShouldMatch) is 3. That is, a document must match at least 3 terms to be considered retrieved.

[0107] 1. Grouping Phase: First, obtain the document frequency (DF) of each word and sort them. Assume the sorted order is ABCDE. According to the formula K=TotalTerms minShouldMatch+1 calculates the number of search terms in the first group to be K=5. 3 + 1 = 3.

[0108] The first group includes Term A, Term B, and Term C.

[0109] The second group includes TermD and TermE.

[0110] 2. Window counting phase: Determine the document identification window (e.g., ID range 0-2047) and establish a counter array CounterMap (i.e., document counter) that matches the document identification window. Traverse the inverted chain of TermA, TermB, and TermC. Assuming that document Doc_100 appears in both TermA and TermB, the value of CounterMap

[100] is 2 (i.e., the number of hits).

[0111] 3. Hybrid Validation and Pruning Phase: Taking document Doc_100 as an example, the hit count for document Doc_100 is 2: First check: Is the number of hits greater than or equal to the recall threshold? If so, recall directly. If not, a second check is required. For example, if the number of hits for Doc_100 is 2 (2 is less than the recall threshold of 3), then a second check (i.e., document recall verification) is performed.

[0112] Second judgment: Calculate the maximum potential value (i.e., recall potential information). The remaining unverified search terms are TermD and TermE, a total of 2. Therefore, 2 (current hit count) + 2 (remaining number of unverified inverted chains) = 4 (4 is greater than or equal to the recall threshold of 3), indicating that the document Doc_100 still has a chance of being recalled and will not be discarded.

[0113] Then, the recall verification is performed through the inverted chain of Terminus: Doc_100 is searched in the inverted chain of Terminus.

[0114] Scenario 1: If TermD is not found (i.e., Doc_100 is not found in the inverted chain of TermD), the hit count for document Doc_100 remains 2. Perform a second check: only TermE (1) remains. 2 (current hit count) + 1 (remaining inverted chains to be verified) = 3 (3 is greater than or equal to the recall threshold of 3), indicating there is still hope for a hit.

[0115] Scenario 2: If TermD is hit (i.e., Doc_100 is found in the inverted chain of TermD), the hit count for document Doc_100 is updated to 3. Then, the first judgment above is executed again: whether 3 (the current hit count) is greater than or equal to 3 (the recall threshold). If so, Doc_100 is recalled directly, and the subsequent search for TermE is terminated.

[0116] Using the embodiments of this application, multiple search terms contained in the search text can be obtained from the query request, and the inverted index of each search term can be obtained. Then, based on the length of the inverted index of each search term, the multiple search terms can be divided into a first group and a second group, wherein the length of the inverted index of the first search term in the first group is less than or equal to the length of the inverted index of the second search term in the second group. Furthermore, based on the documents related to the inverted index of the first search term, the documents in the target document set can be counted for hits. Based on the hit count results, the first recall documents that meet the document recall conditions and the set of documents to be verified that meet the document verification conditions can be selected from the target document set. Then, the documents related to the inverted index of the second search term can be used to perform document recall verification on the set of documents to be verified in the target document set, so as to select the second recall documents that meet the document recall conditions from the set of documents to be verified. Finally, the query results for the search text can be constructed using the first recall documents and the second recall documents. Based on this, the inverted chains of the search terms can be grouped. The first group can be used to count the hits of all documents in the target document set through the inverted chains of the first group. Then, the second group can be used to recall and verify the documents with recall potential (i.e., the set of documents to be verified). This eliminates the need to process all inverted chains of the search terms (especially some long inverted chains) to determine the documents to be recalled, thereby reducing the amount of data processing and improving the efficiency of data query.

[0117] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a data query device provided in an embodiment of this application. For example... Figure 10 As shown, the data query device 1 can be a computer program (including program code) running on a computer device (such as the aforementioned terminal device, such as device 11a). For example, the data query device 1 is an application software; it is understood that the data query device 1 can be used to execute the corresponding steps in the data query method provided in the embodiments of this application. Figure 10 As shown, the data query device 1 may include: a request parsing module 11, a grouping module 12, a first group counting module 13, a recall verification module 14, and a result determination module 15; The request parsing module 11 is used to obtain multiple search terms contained in the search text from the query request, and to obtain the inverted chain of each search term; the inverted chain of each search term is obtained by sorting the document identifiers of the document containing each search term. Grouping module 12 is used to divide multiple search terms into a first group and a second group based on the chain length of the inverted chain of each search term; the chain length of the inverted chain of the first search term in the first group is less than or equal to the chain length of the inverted chain of the second search term in the second group; The first group counting module 13 is used to count the hits of documents in the target document set based on the inverted chain related to the first search term, and to filter out the first recall document that meets the document recall condition and the document verification set that meets the document verification condition based on the hit count result. The recall verification module 14 is used to perform document recall verification on the set of documents to be verified in the target document set through documents related to the inverted chain of the second search term, so as to filter the second recall documents that meet the document recall conditions from the set of documents to be verified. The result determination module 15 is used to construct query results for the retrieved text using the first recall document and the second recall document.

[0118] In one implementation, the target document set includes N documents, where N is a positive integer; The first group counting module 13 is specifically used for: Retrieve the document counter associated with the target document set; the document counter includes N counting positions, and each counting position corresponds to a document in the target document set; If the inverted index of each search term in the first group includes the document identifier of the target document in the target document set, then the cumulative number of times the document identifier of the target document appears in the inverted index of each search term in the first group is determined as the hit count of the target document. The target document's corresponding counting position is determined in the document counter, and the count value at the target document's corresponding counting position is updated to the number of hits for the target document. The hit count result for the target document set is determined based on the count value at each counting position in the document counter.

[0119] In one implementation, the documents in the target document set are all or part of the documents in the document library; each document in the document library has a corresponding document identifier; the hit count result is recorded based on the document counter; The data query device 1 also includes: a window sliding module 16; Window sliding module 16, specifically used for: The document identifiers of each document in the document library are sorted according to their size to obtain a document identifier sequence; The document identifier window is determined from the document identifier sequence according to the preset window size, and the document corresponding to the document identifier in the document identifier window is determined as the target document set; the window size of the document identifier window is the same as the number of counting positions of the document counter; After determining the first and second recalled documents from the target document set based on the document counter associated with the target document set, the document identification window is slid along the document identification sequence according to a preset sliding step size; The target document set is updated based on the document identifiers in the slid-out document identifier window, and the document counter is reset. Based on the reset document counter, the first recall document and the second recall document are determined from the updated target document set, until all document identifiers in the document identifier sequence are used as document identifiers in the document identifier window.

[0120] In one implementation, grouping module 12 is specifically used for: Once the recall threshold associated with the query request is obtained, the number of grouped terms is determined based on the recall threshold and the number of terms for multiple search terms; the number of grouped terms is K, where K is a positive integer. Sort multiple search terms according to the length of the inverted index of each search term; Among multiple search terms, the K search terms with the shortest inverted chain length are identified as the first group, and the search terms other than those in the first group are identified as the second group.

[0121] In one implementation, the hit count includes the number of hits for each document in the target document set; The first group counting module 13 is also used for: Based on the hit count, the first recall document that meets the document recall criteria and the set of documents to be verified that meet the document verification criteria are selected from the target document set, including: The hit count results determine the number of hits for each target document in the target document set. If the number of hits on the target document is greater than or equal to the recall threshold, then the target document is identified as the first recalled document that meets the document recall conditions. If the number of hits of the target document is less than the recall threshold, the recall potential information of the target document is determined based on the number of hits of the target document and the number of second search terms. When the recall potential information of the target document is greater than or equal to the recall threshold, the target document is identified as a document in the set of documents to be verified that meets the document verification conditions.

[0122] In one implementation, the data query device 1 further includes: a discard module 17; Discard module 17, specifically used for: When the recall potential information is less than the recall threshold, the target document is discarded.

[0123] In one implementation, the recall verification module 14 is specifically used for: Determine the documents to be verified from the set of documents to be verified; Determine the validation inverted chain from the inverted chain of the second search term; The document to be verified is recalled and verified based on the verification inverted chain, and the recall verification result is obtained for the document to be verified. The document to be verified is updated based on the document to be verified until every document in the document to be verified is used as a document to be verified for recall verification. The recall verification result includes a first verification result indicating that the document to be verified meets the document recall condition, or a second verification result indicating that the document to be verified meets the document discard condition.

[0124] In one implementation, the recall verification module 14 is further used for: Search for the document identifier of the document to be verified in the verification inverted index; If the document identifier of the document to be verified is found in the verification inverted chain, the number of hits of the document to be verified in the hit count result is accumulated to obtain the update hit count of the document to be verified. If the number of update hits of the document to be verified is greater than or equal to the recall threshold, the first verification result used to indicate that the document to be verified meets the document recall conditions will be used as the recall verification result of the document to be verified.

[0125] In one implementation, the recall verification module 14 is further used for: If the number of update hits of the document to be verified is less than the recall threshold, the recall potential information of the document to be verified is updated based on the number of update hits and the number of inverted chains to be verified, and the updated recall potential information of the document to be verified is obtained; the inverted chain to be verified refers to the inverted chain of the second search term that is not used as a verification inverted chain. When the updated recall potential information of the document to be verified is less than the recall threshold, the first verification result used to indicate that the document to be verified meets the document discarding condition will be used as the recall verification result of the document to be verified, and the document to be verified will be discarded. When the updated recall potential information of the document to be verified is greater than or equal to the recall threshold, the verification inverted chain is updated based on the inverted chain to be verified, and the document to be verified is recalled and verified based on the updated verification inverted chain until the document verification result of the document to be verified is obtained.

[0126] In one implementation, the recall verification module 14 is further used for: If the document identifier of the document to be verified is not found in the verification inverted chain, the verification inverted chain is updated based on the inverted chain in the second search term, and the document to be verified is recalled based on the updated verification inverted chain until the first verification result or the second verification result is obtained.

[0127] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. For example, this computer device can be the terminal device or server described above. Figure 11 As shown, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the processor 1001. Figure 11 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0128] In such Figure 11 In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to execute the data processing method described in any of the corresponding embodiments above, which will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0129] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having a processor performs the methods described in the preceding embodiments. Therefore, these will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0130] The aforementioned computer-readable storage medium can be the apparatus provided in any of the foregoing embodiments or the internal storage unit of the aforementioned computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium may include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0131] Furthermore, it should be noted that this application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The computer program is adapted to be read and executed by a processor, so that a computer device having a processor performs the method provided in any of the corresponding embodiments described above. Additionally, the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer program product or computer program embodiments involved in this application, please refer to the description of the method embodiments of this application.

[0132] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data query method, characterized in that, The method includes: The query request retrieves multiple search terms contained in the search text, and the inverted index of each of the multiple search terms is obtained; the inverted index of each search term is obtained by sorting the document identifiers of the documents containing each search term. Based on the length of the inverted chain of each search term, the multiple search terms are divided into a first group and a second group; the length of the inverted chain of the first search term in the first group is less than or equal to the length of the inverted chain of the second search term in the second group; Based on the inverted chain related documents of the first search term, the documents in the target document set are counted for hits. Based on the hit count results, the first recall document that meets the document recall condition and the document verification set that meets the document verification condition are selected from the target document set. By using documents related to the inverted chain of the second search term, document recall verification is performed on the set of documents to be verified in the target document set, so as to filter the second recall documents that meet the document recall conditions from the set of documents to be verified; The query results for the retrieved text are constructed using the first and second recall documents.

2. The method according to claim 1, characterized in that, The target document set includes N documents, where N is a positive integer; The process of counting the number of documents in the target document set based on the inverted index related to the first search term includes: Obtain a document counter associated with the target document set; the document counter includes N counting positions, and each counting position corresponds to a document in the target document set; If the inverted index of each search term in the first group includes the document identifier of the target document in the target document set, then the cumulative number of times the document identifier of the target document appears in the inverted index of each search term in the first group is determined as the hit count corresponding to the target document. The counting position corresponding to the target document is determined in the document counter, and the count value at the counting position corresponding to the target document is updated to the hit count corresponding to the target document, so as to determine the hit count result corresponding to the target document set based on the count value at each counting position in the document counter.

3. The method according to claim 1, characterized in that, The documents in the target document set are all or part of the documents in the document library; each document in the document library has a corresponding document identifier; the hit count result is recorded based on the document counter; The method further includes: The document identifiers of each document in the document library are sorted according to their size to obtain a document identifier sequence; According to a preset window size, a document identifier window is determined from the document identifier sequence, and the document corresponding to the document identifier in the document identifier window is determined as the target document set; the window size of the document identifier window is the same as the number of counting positions of the document counter; After determining the first and second recalled documents from the target document set based on the document counter associated with the target document set, the document identifier window is slid along the document identifier sequence according to a preset sliding step size; The target document set is updated based on the document identifiers in the slid-out document identifier window, and the document counter is reset. Based on the reset document counter, the first recall document and the second recall document are determined from the updated target document set, until all document identifiers in the document identifier sequence are used as document identifiers in the document identifier window.

4. The method according to claim 1, characterized in that, The process of dividing the multiple search terms into a first group and a second group based on the length of the inverted index of each search term includes: Obtain the recall threshold associated with the query request, and then determine the number of grouped words based on the recall threshold and the number of words of the multiple search terms; the number of grouped words is K, where K is a positive integer; The search terms are sorted according to the length of the inverted index of each search term. Among the multiple search terms, the K search terms with the smallest inverted chain length are determined as the first group, and the search terms other than the first group are determined as the second group.

5. The method according to claim 1, characterized in that, The hit count result includes the number of hits for each document in the target document set; The first recall document that meets the document recall criteria and the set of documents to be verified that meet the document verification criteria are selected from the target document set based on the hit count result, including: The number of hits corresponding to the target documents in the target document set is determined from the hit count results; If the number of hits on the target document is greater than or equal to the recall threshold, then the target document is determined as the first recalled document that meets the document recall conditions; If the number of hits of the target document is less than the recall threshold, the recall potential information of the target document is determined based on the number of hits of the target document and the number of the second search terms. When the recall potential information of the target document is greater than or equal to the recall threshold, the target document is determined as a document in the document set to be verified that meets the document verification conditions.

6. The method according to claim 5, characterized in that, The method further includes: When the recall potential information is less than the recall threshold, the target document is discarded.

7. The method according to claim 1, characterized in that, The step of retrieving and verifying the set of documents to be verified in the target document set by using documents related to the inverted chain of the second search term includes: The documents to be verified are determined from the set of documents to be verified; Determine the verification inverted chain from the inverted chain of the second search term; Based on the verification inverted index, the documents to be verified are recalled and verified to obtain the recall verification results for the documents to be verified. The documents to be verified are then updated based on the set of documents to be verified until each document in the set of documents to be verified is used as a document to be verified for recall verification. The recall verification results include a first verification result indicating that the document to be verified meets the document recall condition, or a second verification result indicating that the document to be verified meets the document discard condition.

8. The method according to claim 7, characterized in that, The step of performing recall verification on the document to be verified based on the verification inverted chain to obtain the recall verification result for the document to be verified includes: Search for the document identifier of the document to be verified in the verification inverted index; If the document identifier of the document to be verified is found in the verification inverted chain, the number of hits of the document to be verified in the hit count result is accumulated to obtain the update hit count of the document to be verified. If the number of update hits of the document to be verified is greater than or equal to the recall threshold, then the first verification result used to indicate that the document to be verified meets the document recall condition will be used as the recall verification result of the document to be verified.

9. The method according to claim 8, characterized in that, The method further includes: If the number of update hits of the document to be verified is less than the recall threshold, the recall potential information of the document to be verified is updated based on the number of update hits and the number of inverted chains to be verified, to obtain the update recall potential information of the document to be verified; the inverted chain to be verified refers to the inverted chain in the inverted chain of the second search term that is not used as the verification inverted chain. When the update recall potential information of the document to be verified is less than the recall threshold, the first verification result used to indicate that the document to be verified meets the document discarding condition is used as the recall verification result of the document to be verified, and the document to be verified is discarded. When the updated recall potential information of the document to be verified is greater than or equal to the recall threshold, the verification inverted chain is updated based on the inverted chain to be verified, and the document to be verified is recalled and verified based on the updated verification inverted chain until the document verification result of the document to be verified is obtained.

10. The method according to claim 8, characterized in that, The method further includes: If the document identifier of the document to be verified is not found in the verification inverted list, the recall potential information of the document to be verified is updated based on the number of hits of the document to be verified and the number of verification inverted lists, so as to obtain the updated recall potential information of the document to be verified. The verification inverted list refers to the inverted list of the second search term that is not used as the verification inverted list. When the update recall potential information of the document to be verified is less than the recall threshold, the first verification result used to indicate that the document to be verified meets the document discarding condition is used as the recall verification result of the document to be verified, and the document to be verified is discarded. When the updated recall potential information of the document to be verified is greater than or equal to the recall threshold, the verification inverted chain is updated based on the inverted chain to be verified, and the document to be verified is recalled and verified based on the updated verification inverted chain until the document verification result of the document to be verified is obtained.

11. A data query device, characterized in that, The device includes: The request parsing module is used to obtain multiple search terms contained in the search text from the query request, and to obtain the inverted list of each of the multiple search terms; the inverted list of each search term is obtained by sorting the document identifiers of the documents containing each search term. The grouping module is used to divide the multiple search terms into a first group and a second group based on the chain length of the inverted chain of each search term; the chain length of the inverted chain of the first search term in the first group is less than or equal to the chain length of the inverted chain of the second search term in the second group; The first group counting module is used to count the hits of documents in the target document set based on the inverted chain related to the first search term, and to filter out the first recall document that meets the document recall condition and the document verification set that meets the document verification condition from the target document set based on the hit count result. The recall verification module is used to perform document recall verification on the set of documents to be verified in the target document set through documents related to the inverted chain of the second search term, so as to filter the second recall documents that meet the document recall conditions from the set of documents to be verified. The result determination module is used to construct query results for the retrieved text using the first recall document and the second recall document.

12. A computer device, characterized in that, Including memory and processor; The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to invoke the computer programs so that the computer device performs the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-10.

14. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, the computer program being adapted to be read and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-10.

Images