A method, apparatus, device, and medium for knowledge base document reordering based on a large language model.

Abstract

This application provides a knowledge base document reordering method, apparatus, device, and medium based on a large language model, relating to the field of data processing, for improving the accuracy of document ranking. The method includes: retrieving multiple candidate documents from a target database based on query information input by a target user; calculating the document relevance between each candidate document and the query information based on multiple document attributes and at least one query field of each candidate document; wherein at least one query field is related to the query intent of the query information, and the document relevance is used to indicate the degree of matching between each candidate document and the query intent indicated by the query information; calculating the comprehensive relevance of each candidate document based on its document relevance and initial relevance; ranking the multiple candidate documents based on their comprehensive relevance, and displaying the ranked candidate documents.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and to, but is not limited to, a method, apparatus, device, and medium for reordering knowledge base documents based on a large language model. Background Technology

[0002] With the rapid development of information technology, databases, as efficient tools for managing massive amounts of data, have been widely used in various fields. Users can retrieve the data they need from the database by entering query information. However, in document retrieval scenarios, the query information entered by users is often quite general. In this case, the system usually displays multiple documents related to the query information for the user to filter out the target document.

[0003] Currently, to present users with documents that better match their query needs, a document ranking scheme has been proposed: a data reordering model is used to sort the multiple documents retrieved from the query, prioritizing those that better match the user's query requirements. However, because the data reordering model has a relatively superficial understanding of the user's true query intent, the ranking results are not accurate. Summary of the Invention

[0004] This application provides a method, apparatus, device, and medium for reordering knowledge base documents based on a large language model, to improve the accuracy of document sorting. The method, apparatus, device, and medium for reordering knowledge base documents based on a large language model provided in this application are implemented as follows: In a first aspect, embodiments of this application provide a knowledge base document reordering method based on a large language model, comprising: retrieving multiple candidate documents from a target database based on query information input by a target user, wherein the target database includes at least two types of databases such as graph databases, text databases, vector databases, and online databases; calculating the document relevance between each candidate document and the query information based on multiple document attributes and at least one query field of each candidate document; wherein the at least one query field is related to the query intent of the query information, and the document relevance is used to indicate the degree of matching between each candidate document and the query intent indicated by the query information; calculating the comprehensive relevance of each candidate document based on its document relevance and initial relevance for each candidate document; wherein the initial relevance is obtained by evaluating each candidate document using a pre-trained reordering model; and sorting the multiple candidate documents based on their comprehensive relevance and displaying the sorted multiple candidate documents.

[0005] In one possible implementation, calculating the document relevance between each candidate document and the query information based on multiple document attributes and at least one query field of each candidate document includes: determining at least one target document attribute from multiple document attributes of each candidate document based on the document attribute indicated by each query field in the at least one query field; calculating the document sub-relevance corresponding to each target document attribute based on the at least one target document attribute and the target weight value corresponding to each target document attribute; and calculating the document relevance between each candidate document and the query information based on the document sub-relevance corresponding to each target document attribute.

[0006] In one possible implementation, the plurality of document attributes includes at least one non-target document attribute, which is used to indicate the document quality of each candidate document. The step of calculating the comprehensive relevance of each candidate document based on its document relevance and initial relevance, for each candidate document among the plurality of candidate documents, includes: calculating the quality sub-relevance corresponding to each non-target document attribute based on the parameter value indicated by the at least one non-target document attribute and the target weight value corresponding to each non-target document attribute; calculating the quality relevance corresponding to the at least one non-target document attribute based on the quality sub-relevance corresponding to each non-target document attribute; and calculating the comprehensive relevance of each candidate document based on its document relevance, the initial relevance, and the quality relevance.

[0007] In one possible implementation, the at least one non-target document attribute includes a knowledge overlap attribute, a document publication attribute, and a user rating attribute, wherein the document publication attribute includes the document effective time attribute, document publication freshness attribute, and publishing unit attribute of each candidate document; the step of calculating the quality sub-relevance corresponding to each non-target document attribute based on the parameter values ​​indicated by the at least one non-target document attribute and the target weight value corresponding to each non-target document attribute includes: for the knowledge overlap attribute, calculating the quality sub-relevance corresponding to the knowledge overlap attribute based on the knowledge overlap between each candidate document and other documents and the target weight value corresponding to the knowledge overlap attribute; for the document publication attribute, calculating the quality sub-relevance corresponding to the document publication attribute based on the publication attribute matching degree between each candidate document and the query information and the target weight value corresponding to the document publication attribute; for the user rating attribute, calculating the quality sub-relevance corresponding to the user rating attribute based on the user rating of each candidate document and the target weight value corresponding to the user attribute rating.

[0008] In one possible implementation, the method further includes: acquiring a historical retrieval information database of a target user group and retrieval score information corresponding to each historical retrieval information, wherein the target user group consists of users who use the hybrid database for document retrieval, and the target user group includes the target user; the retrieval score information corresponding to each historical retrieval information is used to indicate the ranking accuracy of each historical retrieval information for multiple historical candidate documents; classifying the target user group according to the historical query information corresponding to each historical retrieval information in the historical retrieval information database and a preset classification rule to obtain at least one user subset; for each user subset, determining at least one document attribute to be optimized according to the retrieval score information corresponding to each historical retrieval information of each user subset, wherein the at least one document attribute to be optimized is a document attribute that affects the ranking accuracy of each historical retrieval information for multiple historical candidate documents; and adjusting the preset weight value corresponding to each document attribute to be optimized according to the retrieval score information corresponding to the at least one document attribute to be optimized in each user subset to obtain the target weight value corresponding to each document attribute to be optimized.

[0009] In one possible implementation, the retrieval scoring information includes an evaluation score for the ranking accuracy of multiple historical candidate documents corresponding to each historical retrieval information, and each historical retrieval information also includes the historical comprehensive relevance of multiple historical candidate documents corresponding to each historical retrieval information; determining at least one document attribute to be optimized based on the retrieval scoring information corresponding to each historical retrieval information of each user subset includes: filtering an abnormal historical retrieval information set from all historical retrieval information corresponding to each user subset based on the evaluation score and historical comprehensive relevance of each historical retrieval information; filtering the at least one document attribute to be optimized from the multiple document attributes based on the multiple sub-relevances included in the comprehensive relevance of each historical candidate document in each abnormal historical retrieval information, wherein one sub-relevance corresponds to one document attribute.

[0010] In one possible implementation, the method further includes: obtaining a set of historical search information of the target user and search rating information corresponding to each historical search information in the set of historical search information; determining at least one document attribute to be optimized for the target user based on the search rating information corresponding to each historical search information in the set of historical search information of the target user; and adjusting a preset weight value corresponding to each document attribute to be optimized in the at least one document attribute to be optimized for the target user and the corresponding search rating information to obtain a target weight value corresponding to each document attribute to be optimized.

[0011] Secondly, embodiments of this application provide a document processing apparatus, comprising: a retrieval module, configured to retrieve multiple candidate documents from a target database based on query information input by a target user, wherein the target database includes at least two types of databases selected from graph databases, text databases, vector databases, and online databases; a calculation module, configured to calculate a document relevance degree between each candidate document and the query information based on multiple document attributes and at least one query field of each candidate document; wherein the at least one query field is related to the query intent of the query information, and the document relevance degree is used to indicate the degree of matching between each candidate document and the query intent indicated by the query information; the calculation module is further configured to calculate a comprehensive relevance degree for each candidate document among the multiple candidate documents, based on the document relevance degree and initial relevance degree of each candidate document; wherein the initial relevance degree is obtained by evaluating each candidate document through a pre-trained reordering model; and a sorting module, configured to sort the multiple candidate documents based on the comprehensive relevance degree of each candidate document and display the sorted multiple candidate documents.

[0012] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described in the first aspect and any possible implementation.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in the first aspect and any possible implementation.

[0014] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in the first aspect and any possible implementation.

[0015] By implementing the knowledge base document reordering method, apparatus, device, and medium based on a large language model provided in this application, after retrieving multiple candidate documents from a target database based on user query information, the matching degree—document relevance—between each candidate document and the query intent is further determined based on multiple document attributes and at least one query field. The document relevance is then combined with the initial relevance of the reordering model for multiple candidate documents to rank them. Since at least one query field is related to the query intent, the document relevance calculated from at least one query field and multiple document attributes can effectively quantify the matching degree between multiple candidate documents and the query intent. Furthermore, by combining the initial relevance of the reordering model for multiple candidate documents, the model's judgment on basic document relevance can be preserved, and potential biases in the reordering model can be corrected through intent matching, thereby improving the accuracy of document ranking. Moreover, it also facilitates the priority display of candidate documents that meet the user's query needs, thereby improving the user's document retrieval efficiency and user experience. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0017] Figure 1 This application provides an illustration of an application scenario for a knowledge base document reordering method based on a large language model, as shown in the embodiments of this application. Figure 2 A flowchart illustrating a knowledge base document reordering method based on a large language model, provided for embodiments of this application; Figure 3 This is a schematic diagram of an interface for displaying multiple candidate documents after sorting in a query interface, provided as an embodiment of this application. Figure 4 A schematic diagram of a process for calculating comprehensive relevance is provided for an embodiment of this application; Figure 5 This application provides a schematic diagram of a process for adjusting and obtaining target weight values ​​corresponding to document attributes. Figure 1 ; Figure 6 This application provides a schematic diagram of a process for adjusting and obtaining target weight values ​​corresponding to document attributes. Figure 2 ; Figure 7 A schematic diagram of an interface for creating a new sensitive word database is provided for an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a document processing device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] In document retrieval scenarios, user-input queries are typically quite general. Therefore, multiple documents related to the query are usually displayed to allow users to filter for the target document. To present documents that better match the user's query needs, a document ranking scheme has been proposed: a data reordering model sorts the retrieved documents to prioritize those that best match the user's query. However, because the data reordering model only superficially understands the user's true query intent and fails to deeply analyze the user's underlying needs, the ranking results are often inaccurate.

[0021] To address the aforementioned issues, embodiments of this application provide a knowledge base document reordering method based on a large language model, the method comprising: Based on the query information input by the target user, multiple candidate documents are retrieved from the target database, which includes at least two of the following: graph database, text database, vector database, and online database. The document relevance between each candidate document and the query information is calculated based on multiple document attributes and at least one query field of each candidate document. At least one query field is related to the query intent of the query information, and the document relevance indicates the degree of matching between each candidate document and the query intent indicated by the query information. For each candidate document among the multiple candidate documents, a comprehensive relevance is calculated based on the document relevance and the initial relevance. The initial relevance is obtained by evaluating each candidate document using a pre-trained reordering model. Based on the comprehensive relevance of each candidate document, the multiple candidate documents are ranked, and the ranked candidate documents are displayed.

[0022] In this embodiment, since at least one query field is related to the query intent, the document relevance calculated from at least one query field and multiple document attributes can effectively quantify the matching degree between multiple candidate documents and the query intent. Furthermore, by combining the initial relevance of multiple candidate documents by the re-ranking model, the model's judgment on basic document relevance can be preserved, while potential biases in the re-ranking model can be corrected through intent matching, thereby improving the accuracy of document ranking. Moreover, it also facilitates prioritizing the display of candidate documents that meet the user's query needs, thus improving the user's document retrieval efficiency and user experience.

[0023] The following examples illustrate the application scenarios of the knowledge base document reordering method based on a large language model as described in the embodiments of this application.

[0024] Please refer to Figure 1 This is a schematic diagram illustrating an application scenario of a knowledge base document reordering method based on a large language model, provided in an embodiment of this application. Figure 1 As shown in the diagram, this application scenario includes a target user 110, an electronic device 120, and a target database 130. A communication connection is established between the electronic device 120 and the target database 130. The target database 130 also includes databases 131, 132, 133, and 134.

[0025] Wherein, electronic device 120 can refer to a device with data retrieval and data processing capabilities, such as a terminal device or server. Terminal devices include mobile phones, personal computers (PCs), tablets, laptops, handheld computers, mobile internet devices (MIDs), etc. Target database 130 includes at least two databases selected from graph databases, text databases, vector databases, and online databases. For example, Figure 1This example uses a target database 130 comprising a graph database 131, a text database 132, a vector database 133, and an online database 134. However, in practical applications, the target database 130 may include fewer or more databases, and this embodiment does not limit this. A graph database is a database used to store and query graph-structured data, representing data and its relationships through nodes and edges. In this embodiment, the graph database may store knowledge graphs about massive amounts of documents. A text database is a database where users store and manage various forms of text data. In this embodiment, data can be stored in the text database as documents, or documents can be split into multiple text data and stored in the text database. A vector database is a database specifically used to store and query vectors, storing vectors derived from the vectorization of text, speech, images, video, etc. In this embodiment, the vector database is mainly used to store the vectorization of text. An online database, also known as an online database, refers to an internet-based database system where users can access and manipulate data through a network connection.

[0026] For example, target user 110 inputs query information into electronic device 120. Based on the query information, electronic device 120 retrieves multiple candidate documents from the target database, sorts these multiple candidate documents, and then displays the sorted multiple candidate documents to target user 110. The specific methods of retrieval and sorting by electronic device 120 will be described in detail below.

[0027] It should be noted that the knowledge base document reordering method based on a large language model described in this application embodiment can be applied to various scenarios with document retrieval needs, such as government information retrieval scenarios, academic literature retrieval scenarios, enterprise internal document management scenarios, news and information retrieval scenarios, etc., and this application embodiment does not limit it.

[0028] After introducing the background technology and application scenarios of the embodiments of this application, the specific implementation methods of the knowledge base document reordering method based on a large language model provided by the embodiments of this application will be described in detail below.

[0029] Please refer to Figure 2 This is a flowchart illustrating a knowledge base document reordering method based on a large language model, provided in an embodiment of this application. The following description refers to the execution of this method using a document processing device. Figure 2 The steps shown are illustrated by example, and the document processing device is, for example, Figure 1 The electronic device 120 shown.

[0030] S201, based on the query information input by the target user, retrieve multiple candidate documents from the target database, wherein the target database includes at least two of the following: graph database, text database, vector database, and online database.

[0031] The query information is used to indicate the target document that meets the query needs of the target user. For example, in the scenario of government information retrieval, if the query information is to query the 2025 new energy policy, then the target document is the 2025 new energy policy document.

[0032] In some embodiments, the document processing device can obtain query information by acquiring the query text entered by the target user on its display interface. Alternatively, the document processing device can also receive query information sent from a data query device, etc. This application embodiment does not limit this, and the data query device may be, for example, the target user's user equipment, or other devices with data query and data sending capabilities, which this application embodiment does not limit.

[0033] After obtaining the query information, the document processing device can retrieve multiple candidate documents from the target database based on the query information. Since the target database includes at least two types of databases, and the query methods for each type of database are different, the following describes the specific methods by which the document processing device retrieves multiple candidate documents from four types of databases: graph databases, text databases, vector databases, and online databases.

[0034] 1. Graph Database The document processing device can parse at least one query field from the query information and retrieve at least one candidate document from the graph database based on this at least one query field.

[0035] In some embodiments, the document processing apparatus may parse at least one query field from query information based on natural language understanding technology. For example, using analysis and stop word techniques, the query information is first segmented into basic words, then meaningless stop words are filtered out, and the remaining words constitute at least one query field. As another example, the document processing apparatus may extract at least one query field from the query information based on keyword extraction or named entity recognition. Furthermore, the document processing apparatus may input the query information into a pre-trained large language model to obtain at least one query field. The embodiments of this application do not limit the method for parsing to obtain at least one query field.

[0036] Documents in graph databases typically exist as document nodes (representing document entities, such as articles, reports, etc.). Graph databases also include entity nodes related to document attribute information. An entity node indicates a document attribute, and edges between document nodes and entity nodes indicate the relationships between them. Document attribute information can include metadata and content attributes. Metadata includes attributes such as author, document type, document title, publication date, user rating, and publishing institution. Content attributes include keywords, summary, and document topic. In this case, the document processing device can traverse all nodes in the graph database based on at least one query field, querying for candidate document nodes that match all at least one query field, thus obtaining at least one candidate document node. Each candidate document node corresponds to one candidate document, and the document corresponding to these at least one candidate document node is thus considered at least one candidate document.

[0037] For example, taking the query "Query technical documents published in 2024 with the title containing 'artificial intelligence'" as an example, the corresponding at least one query field is "2024", "artificial intelligence", and "technical document". Here, "2024" corresponds to the document publication time attribute, "artificial intelligence" corresponds to the title attribute, and "technical document" corresponds to the document type attribute. Based on these query fields and their corresponding attributes, the document processing device traverses all nodes in the graph database, filtering out at least one candidate document node that meets the criteria of a document publication time of "2024", a title attribute containing "artificial intelligence", and a document type of "technical document", thereby obtaining at least one candidate document.

[0038] 2. Text database The document processing device can parse at least one query field from the query information and retrieve at least one candidate document from the graph database based on this at least one query field.

[0039] The method by which the document processing device obtains at least one query field can be referred to the content described above, and will not be repeated here.

[0040] The document processing device can categorize the at least one query field into a first type of query field and a second type of query field based on the document attributes corresponding to each query field. The document attributes corresponding to the first type of query field are the document's metadata information, while the document attributes corresponding to the second type of query field are the document's content attributes. Based on this, the document processing device can filter at least one candidate document from the text database that matches both the first type of query field and the second type of query field.

[0041] For example, taking a query field that includes at least "Zhang A", "2023", and "blockchain" as an example, where "Zhang A" corresponds to the document's author attribute, "2023" corresponds to the document's publication time attribute, and "blockchain" corresponds to the document's content attribute. The document processing device can then filter an initial set of documents from the text database based on metadata, matching "Zhang A" as the author and "2023" as the publication time. From this initial set of documents, it can then filter out documents whose content contains "blockchain", thus obtaining at least one candidate document.

[0042] It should be noted that the document processing device can also obtain at least one candidate document from the text database through semantic retrieval, keyword combination retrieval, etc., which will not be exemplified one by one in the embodiments of this application.

[0043] 3. Vector Database The document processing device can vectorize query information to obtain query vectors. For example, the document processing device can first preprocess the query information to obtain preprocessed query information. Preprocessing includes data cleaning, word segmentation, stop word removal, etc. Furthermore, the document processing device can use an embedding model to convert the preprocessed query information into query vectors.

[0044] The document processing device calculates the similarity between the query vector and each document vector in the vector database, and identifies documents in the vector database whose similarity to the query vector is greater than or equal to the target similarity as candidate documents, thereby obtaining at least one candidate document. The similarity can be cosine similarity, Euclidean distance, etc., and this embodiment does not limit this. The target similarity can be pre-configured in the document processing device and can be set according to actual needs, for example, a target similarity of 80%, 90%, etc., and this embodiment does not limit this.

[0045] It should be noted that the documents stored in the vector database are all stored in vector form. Document vectors are generated as entire documents, or as paragraphs or chapters. This application embodiment does not limit this. It should be understood that the vectorization processing method for documents in the vector database is the same as that for query information.

[0046] 4. Networked database The document processing device can first establish a network connection with a networked database, then convert the query information into a structured query statement, and retrieve at least one candidate document from the networked database based on the structured query statement. The specific method of converting the query information into a structured query statement and retrieving at least one candidate document from the networked database based on the structured query statement can refer to traditional database query methods, and this application embodiment does not limit this approach.

[0047] Based on the document retrieval operations performed on each database as described above, a total of multiple candidate documents were obtained.

[0048] In some embodiments, because there is overlap in the documents stored in the target database, the candidate documents retrieved from different databases based on the same query information may overlap. Therefore, in order to reduce the subsequent computational load and improve data processing efficiency, the document processing device can perform deduplication on the documents retrieved from the target database to obtain multiple candidate documents, all of which are different documents.

[0049] S202, based on multiple document attributes and at least one query field of each candidate document, calculate the document relevance between each candidate document and the query information; wherein, at least one query field is related to the query intent of the query information, and the document relevance is used to indicate the degree of matching between each candidate document and the query intent indicated by the query information.

[0050] The method for obtaining at least one query field and the description of document attributes can be referred to the content described above, and will not be repeated here. Optionally, when at least one query field is obtained through a pre-trained large language model, the pre-trained large language model can also accurately understand the query intent implied in the query information, and thus extract at least one more accurate query field. The pre-trained large language model can be trained through a large amount of sample query information and sample query fields corresponding to the sample query information. The specific training method can be referred to the traditional supervised learning method, and will not be repeated here in the embodiments of this application.

[0051] In this embodiment of the application, the document processing device can determine the number of document attributes that match the document attributes corresponding to at least one query field among the multiple document attributes of each candidate document, and calculate a score based on the number of matching document attributes, thereby calculating the document relevance between each candidate document and the query information.

[0052] For example, taking a query field that includes "2024", "artificial intelligence", and "technical document" as an example, the document processing device can determine that "2024" corresponds to the document publication time attribute, "artificial intelligence" corresponds to the document topic attribute, and "technical document" corresponds to the document type attribute. Further, the document processing device determines whether the document publication time attribute corresponding to each candidate document is "2024", whether the topic attribute is "artificial intelligence", and whether the document type is "technical document". If candidate document A has a publication time of "2024" and a topic of "artificial intelligence", but its document type is news information, then the number of document attributes that match the document attributes corresponding to at least one query field for candidate document A is 2. Taking a document attribute score of 10 as an example, the document relevance between candidate document A and the query information is 20 points. It should be noted that in practical applications, the specific score setting can be set according to actual needs, and this embodiment does not limit this.

[0053] In one possible implementation, the document processing device may further determine at least one target document attribute from multiple document attributes of each candidate document based on the document attributes indicated by each query field in at least one query field. The target document attribute is the document attribute indicated by the query field, and may be a document's metadata attribute, content attribute, etc., which is not limited in this embodiment. For example, if at least one query field includes "2024," "artificial intelligence," and "technical document," the document processing device may determine that "2024" corresponds to the document publication time attribute, "artificial intelligence" corresponds to the document content attribute, and "technical document" corresponds to the document type attribute. Accordingly, at least one target document attribute includes a document publication time attribute, a topic attribute, and a document type attribute.

[0054] Further, based on at least one target document attribute and the target weight value corresponding to each target document attribute, the document sub-relevance corresponding to each target document attribute is calculated. For example, if the document processing device determines that the attribute value corresponding to each target document attribute is the same as the query field corresponding to each target document attribute, a corresponding score can be obtained. The document sub-relevance corresponding to each target document attribute can be calculated by multiplying this score by the target weight value corresponding to each target document attribute. The target weight value corresponding to each of the at least one target document attributes may be pre-configured in the document processing device. The target weight values ​​corresponding to each target document attribute may be the same or different, and can be set according to actual needs; this embodiment does not limit this.

[0055] For example, taking a query field that includes at least "2024", "artificial intelligence", and "technical document", the document processing device can determine that "2024" corresponds to the document publication time attribute, "artificial intelligence" corresponds to the document content attribute, and "technical document" corresponds to the document type attribute. The target weight value for the document publication time attribute is 0.7, the target weight value for the content attribute is 1, and the target weight value for the document type attribute is 0.9. If candidate document A has a publication time of "2024" and a document topic of "artificial intelligence", but its document type is news, then candidate document A has a score of 10 points for the document publication time attribute. Furthermore, based on the target weight value of 0.7 for the document publication time attribute, the document sub-relevance corresponding to the document publication time attribute is calculated to be 7 points. Candidate document A has a score of 10 points for the content attribute. Furthermore, based on the target weight value of 1 for the content attribute, the document sub-relevance corresponding to the content attribute is calculated to be 10 points. The candidate document has a score of 0 points for the document type attribute, and therefore, the document sub-relevance corresponding to the document type attribute is 0 points.

[0056] Finally, the document processing device can calculate the document relevance between each candidate document and the query information based on the document sub-relevance corresponding to each target document attribute.

[0057] For example, the document processing device determines the document relevance between each candidate document and the query information by summing the document sub-relevances corresponding to at least one target document attribute. As another example, the document processing device determines the document relevance between each candidate document and the query information by weighted summing of the document sub-relevances corresponding to at least one target document attribute. It should be noted that the above methods for calculating document relevance are examples provided in this application's embodiments. In practical applications, document relevance can also be calculated through multiplication or other modified calculation methods, and this application's embodiments do not limit this approach.

[0058] For example, continuing with the example mentioned above, the document relevance between candidate document A and the query information is 10 + 7 + 0 = 17 points.

[0059] In one possible implementation, the document processing device can dynamically adjust the target weight values ​​corresponding to different document attributes based on different document retrieval scenarios or users' historical retrieval information. For example, taking the dynamic adjustment of target weight values ​​corresponding to different document attributes based on different document retrieval scenarios as an example, if the document retrieval scenario is an academic retrieval scenario, users pay more attention to authority and document credibility, so the weight of the corresponding metadata attributes can be set higher (such as document author, issuing unit, etc.); if the document retrieval scenario is a news retrieval scenario, users pay more attention to timeliness and authenticity, so the weight of the corresponding metadata attributes can be set higher (such as document publication time attribute, issuing unit, etc.); if the document retrieval scenario is a question-and-answer system, users pay more attention to whether it can directly solve problems, so the weight of the corresponding content attributes can be set higher; if the document retrieval scenario is a government document retrieval scenario, users pay more attention to the scope of application of policies, so the weight of the corresponding metadata attributes (such as affected area, issuing unit, effective time, etc.) can be set higher.

[0060] In this implementation, the document processing device can identify the current document retrieval scenario based on query information, and determine the target weight value corresponding to each target document attribute in at least one target document attribute based on the identified document retrieval scenario. For example, the document processing device can pre-store weight sets for different document retrieval scenarios, with each weight set including target weight values ​​corresponding to multiple document attributes. Then, after identifying the document retrieval scenario, it can determine the target weight set matching the document retrieval scenario from the multiple weight sets, and determine the target weight value corresponding to each target document attribute in at least one target document attribute based on the target weight set.

[0061] S203. For each candidate document among multiple candidate documents, calculate the comprehensive relevance of each candidate document based on its document relevance and initial relevance; wherein, the initial relevance is obtained by evaluating each candidate document through a pre-trained reordering model.

[0062] The reranking model, also known as the reranker, is used to perform semantic understanding on each candidate document among multiple candidate documents. Based on the semantic understanding results, it scores the similarity between the candidate document and the query information, thereby re-ranking the multiple candidate documents based on the similarity scores. In this embodiment, the pre-trained reranking model can output an initial relevance for each candidate document, which can be understood as the semantic similarity between each candidate document and the query information.

[0063] For example, a document processing device can input multiple candidate documents and query information into a pre-trained rearrangement model to obtain the initial relevance of its output for each candidate document.

[0064] Furthermore, the document processing device can calculate the comprehensive relevance of each candidate document based on its document relevance and initial relevance.

[0065] For example, a document processing device can use the sum of the document relevance of each candidate document and the initial relevance as the comprehensive relevance of each candidate document.

[0066] For example, a document processing device can determine the overall relevance of each candidate document by weighting and summing the document relevance of each candidate document with its initial relevance. One formula for calculating the overall relevance is shown below: Q = a*x + b*y Where Q represents the overall relevance of each candidate document, a represents the weight corresponding to the document relevance, x represents the document relevance of each candidate document, b represents the weight corresponding to the initial relevance, and y represents the initial relevance of each candidate document.

[0067] It should be noted that the values ​​of a and b can be set according to actual needs. For example, a=0.6, b=0.4; a=0.5, b=0.5; a=1, b=1, etc. This application does not limit these values.

[0068] S204: Sort multiple candidate documents according to the overall relevance of each candidate document, and display the sorted candidate documents.

[0069] The document processing device can sort multiple candidate documents in descending order of overall relevance and display the sorted candidate documents.

[0070] For example, if multiple candidate documents include {document1, document2, document3, document4, document5}, after sorting the multiple candidate documents in descending order of overall relevance, the sorted multiple candidate documents include {document2, document4, document3, document1, document5}.

[0071] In one possible implementation, the document processing device can input these multiple candidate documents into a large language model in sequence, so that the large language model can extract document fragments of multiple candidate documents and display them in sequence on the query interface. The document fragments extracted by the large language model can be document summaries of each candidate document, or document fragments of each candidate document with the highest semantic similarity to the query information.

[0072] For example, please refer to Figure 3 This is a schematic diagram illustrating an interface for displaying multiple candidate documents sorted in a query interface, as provided in an embodiment of this application. Figure 3As shown, multiple candidate documents are displayed sequentially in the interface in the order of Document 2, Document 4, Document 3, Document 1, and Document 5. When the area containing Document 2 is selected by the user, the full text of Document 2 is displayed in another area of ​​the interface. Optionally, the area containing each candidate document also displays ranking information based on its overall relevance, as well as its semantic similarity to the query information.

[0073] The ranking method based on comprehensive relevance can accurately identify candidate documents that are more closely matched to the query information among multiple candidate documents, thereby prioritizing the display of more relevant candidate documents to users, improving the user's retrieval efficiency and user experience.

[0074] In one possible implementation, the multiple document attributes include at least one non-target document attribute, which indicates the document quality of each candidate document. In other words, each non-target document attribute is a parameter for measuring document quality. Based on this at least one non-target document attribute, the quality relevance of each candidate document can be calculated. Then, by combining the quality relevance, initial relevance, and document relevance of each candidate document, a comprehensive relevance of each candidate document is calculated. In this way, the ranking result based on the comprehensive relevance is not only closely related to the user's query intent but also has superior document quality, thereby improving the user experience.

[0075] The following is combined Figure 4 This paper explains the specific method for calculating the comprehensive relevance of each candidate document by combining the quality relevance, initial relevance, and document relevance of each candidate document.

[0076] Please refer to Figure 4 This is a schematic diagram of a process for calculating the overall relevance provided in an embodiment of this application.

[0077] S401, calculate the quality sub-relevance of each non-target document attribute based on the parameter value indicated by at least one non-target document attribute and the target weight value corresponding to each non-target document attribute.

[0078] At least one non-target document attribute can refer to a document quality-related attribute other than at least one target document attribute among multiple document attributes. Examples include: knowledge overlap attribute, document publication attribute, and user rating attribute. The knowledge overlap attribute describes the degree of knowledge overlap between the document and other documents. Document publication attributes may include document publication freshness attribute, document issuing unit attribute, document effective date attribute, etc. The document publication freshness attribute indicates the duration of document publication; for example, the shorter the publication duration, the higher the document publication freshness. The document effective date attribute indicates the effective date of the document; for example, if policy document A was published on March 9, 2023, but the policy indicated in policy document A officially took effect on April 1, 2023, then the effective date of policy document A is April 1, 2023. The document issuing unit attribute indicates the authority of the issuing unit. For example, documents published on official websites have higher authority than documents published on ordinary websites.

[0079] It should be noted that document attributes related to document quality may also include other document attributes, which will not be exemplified one by one in this application embodiment.

[0080] The following section uses at least one non-target document attribute, including knowledge overlap, document publication, and user rating, as an example to illustrate how to calculate the quality sub-relevance of each non-target document attribute.

[0081] 1. Knowledge overlap attribute The document processing device can calculate the quality sub-relevance of the knowledge overlap attribute based on the knowledge overlap degree between each candidate document and other documents and the target weight value corresponding to the knowledge overlap degree attribute. In some embodiments, the document processing device can query the target score matching the knowledge overlap degree of each candidate document based on the knowledge overlap degree and the mapping relationship between the preset knowledge overlap degree and the preset score, and calculate the quality sub-relevance of the knowledge overlap attribute of each candidate document based on the target score and the target weight value corresponding to the knowledge overlap attribute. For example, the document processing device can determine the quality sub-relevance of the knowledge overlap attribute of each candidate document as the product of the target score and the target weight value.

[0082] For example, if the knowledge overlap is 30.7%, based on the mapping relationship between the preset knowledge overlap and the preset score, the corresponding target score is determined to be 80 points, and the target weight value corresponding to the knowledge overlap attribute is 0.6, then the quality sub-relevance corresponding to the knowledge overlap attribute is 80*0.6=48 points.

[0083] The following uses candidate document A from a pool of candidate documents as an example to illustrate the specific method for determining the knowledge overlap between each candidate document and other documents in this application embodiment. Here, "other documents" can refer to other candidate documents besides candidate document A from the pool of candidate documents, or other documents in the target database besides candidate document A; this application embodiment does not limit this to any particular document.

[0084] The document processing device can convert candidate document A into a term frequency–inverse document frequency (TF-IDF) vector. Similarly, it can convert each document in other documents into a TF-IDF vector. Then, it can calculate the cosine similarity between the two TF-IDF vectors corresponding to candidate document A and each document in other documents, thus obtaining the knowledge overlap between candidate document A and other documents.

[0085] In another possible implementation, the document processing device can also convert each document in the candidate document A and other documents into semantic vectors using a pre-trained language model, and then calculate the cosine similarity between the two semantic vectors corresponding to each document in the other document, thus obtaining the knowledge overlap between the candidate document A and other documents. The pre-trained language model can be, for example, a word embedding model, a BERT model, etc., and this embodiment of the application does not limit this to a specific model.

[0086] 2. Document publishing attributes The document processing device can calculate the quality sub-relevance of the document publication attribute based on the matching degree of the publication attribute between each candidate document and the query information, and the target weight value corresponding to the document publication attribute. The document publication attribute may include document publication freshness attribute, document publishing unit attribute, document effective time attribute, etc.

[0087] When a document publishing attribute includes multiple document attributes, the quality sub-relevance corresponding to the document publishing attribute can be the sum of the quality sub-relevances of multiple document attributes, or a weighted average result, etc. There can also be other calculation methods, which are not limited in this embodiment.

[0088] For example, taking document publishing attributes including document publishing freshness attribute, document publishing unit attribute, and document effective time attribute as an example, the document processing device can determine the quality sub-relevance of each candidate document in the document publishing attribute dimension by summing the quality sub-relevance of each candidate document in the document publishing attribute, document publishing unit attribute, and document effective time attribute.

[0089] The following sections explain how to calculate the quality sub-relevance of different document attributes.

[0090] (1) Document publication freshness attribute.

[0091] The document processing device can determine the document freshness of each candidate document based on the publication duration between the document's publication time and the target user's query time. This document freshness is essentially the degree of matching between the publication attributes of the candidate document and the query information. The target user's query time is the moment the document processing device obtains the query information. The longer the document's publication duration, the lower its document freshness.

[0092] For example, the document processing device can determine the document publication freshness of each candidate document based on a preset mapping relationship between publication duration and document publication freshness, as well as the publication duration of each candidate document.

[0093] Furthermore, the document processing device can calculate the quality sub-relevance of each candidate document in relation to the document publication freshness attribute based on the document publication freshness of each candidate document and the target weight corresponding to the document publication freshness attribute. For example, the document processing device can determine the quality sub-relevance of each candidate document in relation to the document publication freshness attribute based on the product of the document publication freshness of each candidate document and the target weight corresponding to the document publication freshness attribute.

[0094] (2) Document issuing unit attribute The document processing device can calculate the credibility of each candidate document based on its issuing unit and a preset mapping relationship between issuing units and credibility. Credibility indicates the authority of the issuing unit. It should be understood that different authoritative issuing institutions correspond to different document retrieval scenarios. For example, in a government document retrieval scenario, policy documents published on official websites have higher authority than those from other issuing units. This credibility represents the matching degree of the publication attribute between the candidate document and the queried information.

[0095] Then, based on the credibility of each candidate document in the document publication unit attribute and the corresponding target weight value, the quality sub-relevance of each candidate document in the document publication unit attribute is calculated. For example, the document processing device can determine the quality sub-relevance of each candidate document in the document publication unit attribute by multiplying the credibility of each candidate document in the document publication unit attribute and the corresponding target weight value.

[0096] (3) Document effective time attribute In some government document retrieval scenarios, policy documents that have already taken effect have higher information value for the target user. Therefore, the document processing device can determine whether a candidate document has taken effect based on its effective date and the target user's query time. If the effective date is earlier than the target user's query time, the candidate document is determined to be effective. In some scenarios, the policy document's effective date attribute also includes "expired," and the document's expiration status can be determined based on the document's repeal date. If the effective date is later than the target user's query time, the candidate document is determined to be ineffective. Based on this, the document processing device can determine the status score corresponding to each candidate document based on the document's effective status and the pre-stored mapping relationship between different effective statuses and preset status scores. This status score represents the matching degree of the publication attributes between the candidate document and the query information. For example, the preset status score for "effective" (including "not expired") is 30 points, the preset status score for "ineffective" is 20 points, the preset status score for "expired" is 10 points, and so on. Specific score settings can be set according to actual needs, and this application embodiment does not limit this.

[0097] Furthermore, after calculating the state score of each candidate document, the quality sub-relevance of each candidate document in the document effective time attribute can be calculated based on the state score of each candidate document and the target weight value corresponding to the document effective time attribute.

[0098] For example, the document processing device can determine the quality sub-relevance of each candidate document in the document effective time attribute by multiplying the state score of each candidate document with the target weight value corresponding to the document effective time attribute.

[0099] It should be noted that the target weight values ​​corresponding to the document publication freshness attribute, the document issuing unit attribute, and the document effective time attribute can be the same or different, and can be determined according to actual needs. This application embodiment does not limit this.

[0100] In another possible implementation, the document publication attribute also includes a document status attribute, which indicates the current publication status of the document, such as published (or public) and unpublished (or unpublic). In some document retrieval scenarios, unpublished or unpublicated documents cannot be displayed to the target user. Therefore, based on the document status attribute, unpublished or unpublicated candidate documents can be directly filtered out from multiple candidate documents.

[0101] 3. User rating attributes The document processing device can calculate the quality sub-relevance of the user rating attribute based on the user rating and the target weight value corresponding to the user attribute rating for each candidate document. For example, the document processing device can determine the quality sub-relevance of the user rating attribute for each candidate document by multiplying the user rating and the target weight value corresponding to the user attribute rating for each candidate document.

[0102] The user rating can be calculated based on at least one of the following parameters for each candidate document: number of likes, number of views, number of downloads, number of favorites, and number of shares. The calculation method can be weighted average, summation, etc., and this application embodiment does not limit the method.

[0103] S402, calculate the quality relevance of at least one non-target document attribute based on the quality sub-relevance of each non-target document attribute.

[0104] The document processing device can calculate the sum of the quality sub-relevances corresponding to at least one non-target document attribute based on the quality sub-relevance corresponding to each non-target document attribute, thereby obtaining the quality relevance corresponding to at least one non-target document attribute.

[0105] For example, if at least one non-target document attribute includes a knowledge overlap attribute, a document publication attribute, and a user rating attribute, then the quality relevance corresponding to at least one non-target document attribute is the sum of the quality sub-relevance of the knowledge overlap attribute, the quality sub-relevance of the document publication attribute, and the quality sub-relevance of the user rating attribute.

[0106] S403. Calculate the overall relevance of each candidate document based on its document relevance, initial relevance, and quality relevance.

[0107] The document processing device can calculate the comprehensive relevance of each candidate document based on the sum of its document relevance, initial relevance, and quality relevance.

[0108] Alternatively, the document processing device can calculate the overall relevance of each candidate document based on the weighted average of the sum of the document relevance, initial relevance, and quality relevance of each candidate document.

[0109] For example, this application provides a formula for calculating comprehensive relevance, as shown below: Overall relevance = α * initial relevance + β * document relevance + γ * quality relevance.

[0110] Where α is the weight value corresponding to the initial relevance, β is the weight value corresponding to the document relevance, and γ is the weight value corresponding to the quality relevance.

[0111] Alternatively, the formula for calculating the overall relevance can also be as follows: Overall Relevance = α * Initial Relevance + β1 * Sub-relevance of Target Document Attribute 1 + β2 * Sub-relevance of Target Document Attribute 2 + ... + βn * Sub-relevance of Target Document Attribute 1 + γ1 * Quality Sub-relevance of Non-Target Document Attribute 1 + γ2 * Quality Sub-relevance of Non-Target Document Attribute 1 + ... + γm * Quality Sub-relevance of Non-Target Document Attribute m The document sub-relevance corresponding to the target document attribute m can be calculated based on the score corresponding to each target document attribute as described above. For example, if the parameter value of a target document attribute matches the parameter value indicated by the corresponding query field, a score of 10 can be obtained, which is the document sub-relevance corresponding to the target document attribute m. The quality sub-relevance corresponding to non-target document attributes m can be referred to the content described above, and will not be repeated here. β1, β2...βn can be different, and γ1, γ2...γm can be different; this application embodiment does not limit this.

[0112] The target weight value for each document attribute (target document attribute or non-target document attribute) mentioned above can be pre-configured in the document processing device. The target weight value for each document attribute can be set according to actual needs, and this application embodiment does not limit this. It should be understood that, in order to avoid quality relevance excessively affecting the document ranking results, resulting in the final ranking results not matching the user's query intent, the target weight value of non-target document attributes can be less than the target weight value corresponding to the target document attribute.

[0113] In one possible implementation, to improve the accuracy of the ranking results, the target weight values ​​corresponding to different document attributes can be adjusted according to the search preferences of different user groups, thereby making the calculated comprehensive relevance more in line with the user's query needs. The following section combines... Figure 5 The specific method for adjusting the target weight value corresponding to the document attribute in the embodiments of this application is described.

[0114] Please refer to Figure 5 This is a schematic diagram of a process for adjusting and obtaining the target weight value corresponding to document attributes, provided in an embodiment of this application. Figure 1 .

[0115] S501, Obtain the historical search information database of the target user group and the search score information corresponding to each historical search information. The target user group is the users who use the hybrid database to search for documents. The target user group includes target users. The search score information corresponding to each historical search information is used to indicate the ranking accuracy of each historical search information for multiple historical candidate documents.

[0116] In layman's terms, the target user group can be any user who uses the hybrid database for document retrieval. In some embodiments, since some users use the hybrid database for document retrieval less frequently (e.g., once, three times), and the document processing device cannot accurately analyze these users' retrieval preferences based on this limited historical retrieval information, the target user group can also refer to any user who frequently uses the hybrid database for document retrieval. For example, the target user group can refer to any user who uses the hybrid database for document retrieval more than or equal to a target number of times, or any user whose duration of using the hybrid database for document retrieval is greater than or equal to a target duration. The target number of times and the target duration can be pre-configured in the document processing device and can be set according to actual needs. For example, the target number of times could be 5, 8, 10, etc., and the target duration could be one week, half a month, one month, etc. This application embodiment does not limit this. The hybrid database refers to a database including the text database, graph database, vector database, etc., mentioned above.

[0117] The historical search information database includes the historical search information of each user in the target user group. This historical search information is stored in the historical search information database in units of the number of searches. In other words, when a user uses the hybrid database to perform a document search, a corresponding historical search information is generated and stored in the historical search information database.

[0118] Each historical retrieval record includes retrieval score information, which indicates the ranking accuracy of multiple historical candidate documents in the corresponding historical retrieval record. These multiple historical candidate documents refer to the retrieval results corresponding to the query information in the corresponding historical retrieval record.

[0119] In some embodiments, the retrieval scoring information may include an evaluation score for the ranking accuracy of multiple historical candidate documents corresponding to each historical retrieval information.

[0120] In some embodiments, each historical retrieval information also includes the historical comprehensive relevance of multiple historical candidate documents corresponding to each historical retrieval information. The calculation method for the comprehensive relevance can be referred to the content described above, and will not be repeated here.

[0121] For example, a document processing device may obtain a historical retrieval information database of a target user group from a pre-configured database.

[0122] S502, based on the historical query information corresponding to each historical retrieval information in the historical retrieval information database and the preset classification rules, classify the target user group to obtain at least one user subset.

[0123] The preset classification rules can be determined based on the user's query preferences (or retrieval preferences), such as query domain, query intent, and the dimensions of information they are interested in.

[0124] In some embodiments, the document processing device may analyze the query preferences corresponding to each historical retrieval information in the historical retrieval information database, and based on the query preferences corresponding to each historical retrieval information, divide all historical retrieval information in the historical retrieval information database into at least one user subset.

[0125] For example, using preset classification rules to indicate the user's query domain, users whose query domain is government document retrieval are classified into one subset of users, and users whose query domain is academic document retrieval are classified into another subset of users.

[0126] S503, for each user subset, based on the retrieval score information corresponding to each historical retrieval information of each user subset, determine at least one document attribute to be optimized, wherein the at least one document attribute to be optimized is a document attribute that affects the ranking accuracy of each historical retrieval information for multiple historical candidate documents.

[0127] The retrieval scoring information includes an evaluation score for the ranking accuracy of multiple historical candidate documents corresponding to each historical retrieval information, and each historical retrieval information also includes the historical comprehensive relevance of multiple historical candidate documents corresponding to each historical retrieval information.

[0128] For each user subset, the document processing device can filter out a set of abnormal historical search information from all historical search information corresponding to each user subset based on the evaluation score and historical comprehensive relevance corresponding to each historical search information.

[0129] In some embodiments, the document processing device may filter out abnormal historical retrieval information of abnormal documents, including those with "high rating and low ranking" and those with "low rating and high ranking," from all historical retrieval information corresponding to each user subset, to form an abnormal historical retrieval information set. "High rating and low ranking" refers to a document with a high rating but a low ranking based on overall relevance. "Low rating and high ranking" refers to a document with a low rating but a high ranking based on overall relevance.

[0130] For example, the document processing device can, for each historical retrieval information, identify historical candidate documents whose evaluation scores are greater than or equal to the target evaluation score and whose overall ranking is in the bottom 50% of the overall ranking of the historical candidate documents as abnormal documents; or, identify historical candidate documents whose evaluation scores are less than the target evaluation score and whose ranking is in the top 50% of the overall ranking of the historical candidate documents as abnormal documents. Accordingly, the historical retrieval information including abnormal documents is called abnormal historical retrieval information. The target evaluation score can be pre-configured in the document processing device and can be set according to actual needs, such as 80 points; this embodiment does not limit this.

[0131] Furthermore, the document processing device can filter out the at least one document attribute to be optimized from the multiple document attributes based on the multiple sub-relevance values ​​included in the comprehensive relevance value of each historical candidate document in each abnormal historical retrieval information, wherein one sub-relevance value corresponds to one document attribute. It should be understood that the multiple sub-relevance values ​​mentioned here include document sub-relevance values ​​calculated from the target document attribute and quality sub-relevance values ​​calculated from non-target document attributes.

[0132] In some embodiments, the document processing device can compare the evaluation scores of abnormal documents with the differences between the corresponding sub-relevances to determine the document attributes to be optimized. For each sub-relevance in the abnormal historical retrieval information of "high score, low ranking", each sub-relevance and its corresponding evaluation score are normalized to the same interval, such as [0,1]. Then, the score difference between the normalized evaluation score and the normalized sub-relevance is calculated. If the score difference is greater than or equal to the target value, then the document attribute corresponding to that sub-relevance is the attribute to be optimized.

[0133] For some abnormal historical search information with "high ranking and low score", the document processing device can calculate the difference between the normalized sub-relevance and the normalized evaluation score. If the score difference is greater than or equal to the target value, then the document attribute corresponding to the sub-relevance is the attribute to be optimized.

[0134] Optionally, to improve the accuracy of optimization, the document processing device can calculate the average score difference corresponding to the same sub-relevance in the abnormal historical retrieval information set for each sub-relevance. If the average value is greater than or equal to the target value, then the document attribute corresponding to that sub-relevance is the attribute to be optimized. The target value can be pre-configured in the document processing device and can be set according to actual needs, such as 0.5, 0.6, etc. This embodiment does not limit this.

[0135] In this way, the document processing device can determine at least one attribute to be optimized from multiple document attributes.

[0136] S504, based on the retrieval score information corresponding to at least one document attribute to be optimized in each user subset, adjust the preset weight value corresponding to each document attribute to be optimized to obtain the target weight value corresponding to each document attribute to be optimized.

[0137] For at least one document attribute to be optimized, if the attribute was calculated using a "high score, low ranking" method, the preset weight value corresponding to that attribute can be increased to obtain a target weight value. If the attribute was calculated using a "low score, high ranking" method, the preset weight value corresponding to that attribute can be decreased to obtain a target weight value. The weight can be adjusted according to a preset step size, such as 0.05, 0.1, etc., but this embodiment does not limit the adjustment.

[0138] In one possible implementation, to better adapt to the query needs of each user, the document processing device can also adjust the target weight value corresponding to each document attribute for the target user based on the target user's historical search information set and corresponding search rating information. The following is combined with... Figure 6 The specific method for obtaining the target weight value corresponding to the document attribute through adjustment as described in the embodiments of this application will be explained.

[0139] Please refer to Figure 6 This is a schematic diagram of a process for adjusting and obtaining the target weight value corresponding to document attributes, provided in an embodiment of this application. Figure 2 .

[0140] S601, obtain the target user's historical search information set and the search score information corresponding to each historical search information in the historical search information set.

[0141] The descriptions of historical search information and search rating information can be found in the preceding text and will not be repeated here.

[0142] For example, the document processing device can obtain the target user's identity information and retrieve the target user's historical search information set from the database based on the identity information. Identity information includes, for example, mobile phone numbers, email addresses, account information, etc., but this embodiment does not limit the specific information provided.

[0143] S602, based on the search rating information corresponding to each historical search information in the target user's historical search information set, determine at least one document attribute to be optimized for the target user.

[0144] The specific method for determining at least one document attribute to be optimized based on the target user's historical search information set and corresponding search rating information can be referred to the content described in step S503 above, and will not be repeated here.

[0145] S603, based on at least one document attribute to be optimized corresponding to the target user and the corresponding search rating information, adjust the preset weight value corresponding to each document attribute to be optimized in the at least one document attribute to be optimized of the target user, and obtain the target weight value corresponding to each document attribute to be optimized.

[0146] The specific method of adjusting the preset weight value corresponding to each document attribute in at least one document attribute of the target user to obtain the target weight value can be referred to the content described in step S504 above, and will not be repeated here.

[0147] In one possible implementation, since the amount of historical search information stored in the database for each user is relatively small in the early stages of document retrieval, it is impossible to accurately analyze the query preferences of each user. Therefore, the target weight values ​​of the corresponding document attributes to be optimized can be adjusted based on the classification of the target user group. In the later stages of document retrieval, the database stores rich historical search information for each user, and the document processing device can then adjust the target weight values ​​of the corresponding document attributes to be optimized for each user to improve the accuracy of the ranking results.

[0148] In another possible implementation, the document processing device may also determine the target weight value corresponding to each document attribute through experimentation. For example, by adjusting the target weight value corresponding to a document attribute one by one in turn, the target weight value corresponding to each document attribute may be obtained experimentally; or experiments may be conducted on different combinations of target weight values ​​to obtain the combination of target weight values ​​with the best experimental effect, etc., wherein each combination of target weight values ​​includes target weight values ​​corresponding to multiple document attributes.

[0149] In one possible implementation, before retrieving multiple candidate documents from the target database, the documents need to be stored in a text database, a vector database, and a graph database. For example, taking document 1 as an example, the document processing device can break down document 1 into paragraphs to obtain multiple document fragments, and store these fragments in the text database, vector database, and graph database respectively. Each document fragment is associated with multiple document attributes and at least one keyword. The at least one keyword can be extracted from each document fragment using a deep learning model, such as the BERT model. The at least one keyword is used to indicate the semantic content of the document fragment, and can also be used subsequently to match the semantic similarity between query information and documents.

[0150] Optionally, since the stop words and sensitive words involved may differ for documents in different fields, different stop words and sensitive words can be selected for filtering when extracting keywords for documents in different fields. For example, please refer to... Figure 7 This is a schematic diagram of an interface for creating a new sensitive word database, provided in an embodiment of this application. Figure 7 As shown, the sensitive word database management interface can include multiple sensitive word databases, and each sensitive word database can be built for a different knowledge base, which are document databases for different fields. For example... Figure 7 The image on the right shows an example of creating a new sensitive word database. For instance, a knowledge base for laws and regulations could include sensitive words such as "law" and "regulations." Since the knowledge base contains only documents related to laws and regulations, the words "law" and "regulations" are meaningless keywords in this database and can therefore be set as sensitive words.

[0151] In one possible implementation, since the target document attributes and non-target document attributes involved in the same retrieval scenario are approximately the same, each document fragment can be associated with at least one target document attribute and at least one non-target document attribute during document entry. For example, the target document attribute is identified by a core tag, and the non-target document attribute is identified by a regular tag. Therefore, during subsequent document retrieval, the document processing device can quickly obtain at least one target document attribute and at least one non-target document attribute based on the core tag and the regular tag.

[0152] For example, in a government affairs retrieval scenario, the target document attributes identified by core tags may include: the region of influence, document type, document publication time, issuing unit, etc. Non-target document attributes identified by ordinary tags may include document publication freshness, knowledge overlap, user rating, etc.

[0153] By implementing the document sorting method of this application embodiment, the problem of document sorting errors caused by the inability to accurately identify the user's query intent during document retrieval can be effectively avoided. For example, if the user inputs the query information "policies of the 'Hundred, Thousand, Ten Thousand Project' released by City A", but the final search result is the policy of the "Hundred, Thousand, Ten Thousand Project" of City B, it may be because the policy content of City B is better written, has a higher number of collections and views, and thus was matched. However, this does not match the user's query intent. By using the comprehensive relevance sorting method in this application embodiment, the user's query intent of "City A" can be effectively identified. Therefore, during sorting, the target document attribute score corresponding to "City A" can effectively avoid the problem of other non-target document attribute scores being higher, thus improving the accuracy of the final sorting result.

[0154] Based on the same inventive concept, embodiments of this application provide a document processing apparatus for implementing any of the aforementioned knowledge base document reordering methods based on a large language model, for example... Figure 2 The document reordering method based on a large language model is shown, and the document processing device can also realize the functions of the document processing device mentioned above.

[0155] Please see Figure 8 This is a schematic diagram of the structure of a document processing device provided in an embodiment of this application. Figure 8 As shown, the document processing device 800 includes a retrieval module 801, a calculation module 802, and a sorting module 803.

[0156] For example, the retrieval module 801 is used to retrieve multiple candidate documents from a target database based on query information input by the target user, wherein the target database includes at least two types of databases such as graph databases, text databases, vector databases, and online databases; the calculation module 802 is used to calculate the document relevance between each candidate document and the query information based on multiple document attributes and at least one query field of each candidate document; wherein at least one query field is related to the query intent of the query information, and the document relevance is used to indicate the degree of matching between each candidate document and the query intent indicated by the query information; the calculation module 802 is also used to calculate the comprehensive relevance of each candidate document based on the document relevance and the initial relevance of each candidate document; wherein the initial relevance is obtained by evaluating each candidate document through a pre-trained reordering model; the ranking module 803 is used to rank the multiple candidate documents based on the comprehensive relevance of each candidate document and display the ranked multiple candidate documents.

[0157] In one possible implementation, the calculation module 802 is specifically configured to: determine at least one target document attribute from multiple document attributes of each candidate document based on the document attribute indicated by each query field in at least one query field; calculate the document sub-relevance corresponding to each target document attribute based on the at least one target document attribute and the target weight value corresponding to each target document attribute; and calculate the document relevance between each candidate document and the query information based on the document sub-relevance corresponding to each target document attribute.

[0158] In one possible implementation, the multiple document attributes include at least one non-target document attribute, which is used to indicate the document quality of each candidate document. The calculation module 802 is specifically used to: calculate the quality sub-relevance corresponding to each non-target document attribute based on the parameter value indicated by the at least one non-target document attribute and the target weight value corresponding to each non-target document attribute; calculate the quality relevance corresponding to at least one non-target document attribute based on the quality sub-relevance corresponding to each non-target document attribute; and calculate the comprehensive relevance of each candidate document based on the document relevance, initial relevance, and quality relevance of each candidate document.

[0159] In one possible implementation, at least one non-target document attribute includes a knowledge overlap attribute, a document publication attribute, and a user rating attribute. The document publication attribute includes the document effective time attribute, document publication freshness attribute, and publishing unit attribute for each candidate document. The calculation module 802 is specifically configured to: for the knowledge overlap attribute, calculate the quality sub-relevance corresponding to the knowledge overlap attribute based on the knowledge overlap between each candidate document and other documents and the target weight value corresponding to the knowledge overlap attribute; for the document publication attribute, calculate the quality sub-relevance corresponding to the document publication attribute based on the publication attribute matching degree between each candidate document and the query information and the target weight value corresponding to the document publication attribute; and for the user rating attribute, calculate the quality sub-relevance corresponding to the user rating attribute based on the user rating of each candidate document and the target weight value corresponding to the user attribute rating.

[0160] In one possible implementation, the calculation module 802 is further configured to: acquire a historical retrieval information database of the target user group and retrieval score information corresponding to each historical retrieval information, wherein the target user group consists of users who use a hybrid database for document retrieval, and includes target users; the retrieval score information corresponding to each historical retrieval information is used to indicate the ranking accuracy of each historical retrieval information for multiple historical candidate documents; classify the target user group according to the historical query information corresponding to each historical retrieval information in the historical retrieval information database and a preset classification rule to obtain at least one user subset; for each user subset, determine at least one document attribute to be optimized according to the retrieval score information corresponding to each historical retrieval information of each user subset, wherein the at least one document attribute to be optimized is a document attribute that affects the ranking accuracy of each historical retrieval information for multiple historical candidate documents; and adjust the preset weight value corresponding to each document attribute to be optimized according to the retrieval score information corresponding to the at least one document attribute to be optimized of each user subset to obtain the target weight value corresponding to each document attribute to be optimized.

[0161] In one possible implementation, the retrieval scoring information includes an evaluation score for the ranking accuracy of multiple historical candidate documents corresponding to each historical retrieval information, and each historical retrieval information also includes the historical comprehensive relevance of multiple historical candidate documents corresponding to each historical retrieval information; the calculation module 802 is specifically used to: filter out an abnormal historical retrieval information set from all historical retrieval information corresponding to each user subset based on the evaluation score and historical comprehensive relevance of each historical retrieval information; and filter out at least one document attribute to be optimized from multiple document attributes based on the multiple sub-relevances included in the comprehensive relevance of each historical candidate document in each abnormal historical retrieval information, wherein one sub-relevance corresponds to one document attribute.

[0162] In one possible implementation, the calculation module 802 is further configured to: obtain a set of historical search information of the target user and search score information corresponding to each historical search information in the set of historical search information; determine at least one document attribute to be optimized for the target user based on the search score information corresponding to each historical search information in the set of historical search information of the target user; and adjust the preset weight value corresponding to each document attribute to be optimized in the at least one document attribute to be optimized for the target user and the corresponding search score information to obtain the target weight value corresponding to each document attribute to be optimized.

[0163] Please refer to Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device 900 includes at least one processor 901 and a memory 902 communicatively connected to the at least one processor 901. The processor 901 can be a general-purpose processor or a dedicated processor. For example, the processor 901 may include a baseband processor or a central processing unit (CPU). The baseband processor can be used to process communication protocols and communication data. The CPU can be used to control the electronic device 900, execute software programs, and / or process data. Different processors can be independent devices or can be integrated into one or more processing circuits, for example, integrated on one or more application-specific integrated circuits (ASICs). In one embodiment, memory 902 stores instructions that can be executed by at least one processor 901. At least one processor 901 implements the functions of the aforementioned document processing device by executing the instructions stored in memory 902, and correspondingly, can also implement the steps executed by the aforementioned document processing device. In this embodiment, the electronic device 900 can also perform the functions of the preceding document processing device 800, and at least one processor 901 in the electronic device 900 can also perform the functions of the preceding document retrieval module 801, the calculation module 802 and the sorting module 803.

[0164] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, they cause the computer to perform any of the above-described knowledge base document reordering methods based on a large language model, for example... Figure 2 The document reordering method based on a large language model is shown. Based on the same inventive concept, embodiments of this application provide a computer program product containing computer instructions that, when run on a computer, enable the implementation of any of the aforementioned knowledge base document reordering methods based on a large language model, for example... Figure 2 The document reordering method based on a large language model is shown.

[0165] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0166] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0167] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0168] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0170] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0172] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0173] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0174] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0175] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0176] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0177] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A knowledge base document reordering method based on a large language model, characterized in that, include: Based on the query information input by the target user, multiple candidate documents are retrieved from the target database, wherein the target database includes at least two types of databases such as graph database, text database, vector database, and online database; Based on multiple document attributes and at least one query field for each candidate document, the document relevance between each candidate document and the query information is calculated; wherein, the at least one query field is related to the query intent of the query information, and the document relevance is used to indicate the degree of matching between each candidate document and the query intent indicated by the query information; For each of the plurality of candidate documents, a comprehensive relevance score is calculated based on the document relevance score and initial relevance score of each candidate document; wherein, the initial relevance score is obtained by evaluating each candidate document through a pre-trained reordering model; Based on the overall relevance of each candidate document, the multiple candidate documents are sorted and the sorted candidate documents are displayed. The plurality of document attributes includes at least one non-target document attribute, which is used to indicate the document quality of each candidate document. For each candidate document among the plurality of candidate documents, the comprehensive relevance of each candidate document is calculated based on its document relevance and initial relevance, including: Based on the parameter values ​​indicated by the at least one non-target document attribute and the target weight value corresponding to each non-target document attribute, the quality sub-relevance corresponding to each non-target document attribute is calculated; Based on the quality sub-relevance corresponding to each non-target document attribute, the quality relevance corresponding to the at least one non-target document attribute is calculated; The comprehensive relevance of each candidate document is calculated based on its document relevance, initial relevance, and quality relevance.

2. The method according to claim 1, characterized in that, The step of calculating the document relevance between each candidate document and the query information based on multiple document attributes and at least one query field for each candidate document includes: Based on the document attributes indicated by each of the at least one query field, at least one target document attribute is determined from a plurality of document attributes of each candidate document; The document sub-relevance corresponding to each target document attribute is calculated based on at least one target document attribute and the target weight value corresponding to each target document attribute; Based on the document sub-relevance corresponding to each target document attribute, the document relevance between each candidate document and the query information is calculated.

3. The method according to claim 1, characterized in that, The at least one non-target document attribute includes a knowledge overlap attribute, a document publication attribute, and a user rating attribute, wherein the document publication attribute includes the document effective time attribute, document publication freshness attribute, and publishing unit attribute of each candidate document; the step of calculating the quality sub-relevance corresponding to each non-target document attribute based on the parameter values ​​indicated by the at least one non-target document attribute and the target weight value corresponding to each non-target document attribute includes: For the knowledge overlap attribute, the quality sub-relevance corresponding to the knowledge overlap attribute is calculated based on the knowledge overlap between each candidate document and other documents and the target weight value corresponding to the knowledge overlap attribute. For the document publishing attribute, the quality sub-relevance corresponding to the document publishing attribute is calculated based on the matching degree of the publishing attribute between each candidate document and the query information and the target weight value corresponding to the document publishing attribute; For the user rating attribute, the quality sub-relevance corresponding to the user rating attribute is calculated based on the user rating of each candidate document and the target weight value corresponding to the user rating attribute.

4. The method according to any one of claims 2-3, characterized in that, The method further includes: The system obtains a historical search information database for the target user group and search score information corresponding to each historical search information. The target user group refers to users who use a hybrid database to search for documents. The target user group includes the target users. The search score information corresponding to each historical search information is used to indicate the ranking accuracy of each historical search information for multiple historical candidate documents. Based on the historical query information corresponding to each historical retrieval information in the historical retrieval information database and the preset classification rules, the target user group is classified to obtain at least one user subset; For each user subset, based on the search score information corresponding to each historical search information of each user subset, at least one document attribute to be optimized is determined. The at least one document attribute to be optimized is a document attribute that affects the ranking accuracy of each historical search information for multiple historical candidate documents. Based on the retrieval scoring information corresponding to at least one document attribute to be optimized for each user subset, the preset weight value corresponding to each document attribute to be optimized is adjusted to obtain the target weight value corresponding to each document attribute to be optimized.

5. The method according to claim 4, characterized in that, The retrieval scoring information includes an evaluation score for the ranking accuracy of multiple historical candidate documents corresponding to each historical retrieval information, and each historical retrieval information also includes the historical comprehensive relevance of multiple historical candidate documents corresponding to each historical retrieval information; the step of determining at least one document attribute to be optimized based on the retrieval scoring information corresponding to each historical retrieval information of each user subset includes: Based on the evaluation score and historical comprehensive relevance corresponding to each historical search information, an abnormal historical search information set is obtained by filtering all historical search information corresponding to each user subset. Based on the multiple sub-relevances included in the comprehensive relevance of each historical candidate document in each abnormal historical retrieval information, at least one document attribute to be optimized is obtained from the multiple document attributes, wherein one sub-relevance corresponds to one document attribute.

6. The method according to any one of claims 2-3, characterized in that, The method further includes: Obtain the target user's historical search information set and the search rating information corresponding to each historical search information in the historical search information set; Based on the search rating information corresponding to each historical search information in the target user's historical search information set, determine at least one document attribute to be optimized for the target user; Based on at least one document attribute to be optimized corresponding to the target user and the corresponding search rating information, adjust the preset weight value corresponding to each document attribute to be optimized in the at least one document attribute to be optimized of the target user to obtain the target weight value corresponding to each document attribute to be optimized.

7. A document processing apparatus, characterized in that, include: The retrieval module is used to retrieve multiple candidate documents from a target database based on query information input by the target user. The target database includes at least two types of databases, such as graph databases, text databases, vector databases, and online databases. The calculation module is used to calculate the document relevance between each candidate document and the query information based on multiple document attributes and at least one query field for each candidate document; wherein the at least one query field is related to the query intent of the query information, and the document relevance is used to indicate the degree of matching between each candidate document and the query intent indicated by the query information; The calculation module is further configured to calculate the comprehensive relevance of each candidate document among the plurality of candidate documents, based on the document relevance and initial relevance of each candidate document; wherein the initial relevance is obtained by evaluating each candidate document through a pre-trained reordering model; The sorting module is used to sort the multiple candidate documents according to the comprehensive relevance of each candidate document, and display the sorted multiple candidate documents; The plurality of document attributes include at least one non-target document attribute, which is used to indicate the document quality of each candidate document. The calculation module is specifically used to: for each candidate document among the plurality of candidate documents, calculate the quality sub-relevance corresponding to each non-target document attribute based on the parameter value indicated by the at least one non-target document attribute and the target weight value corresponding to each non-target document attribute. Based on the quality sub-relevance corresponding to each non-target document attribute, the quality relevance corresponding to the at least one non-target document attribute is calculated; The comprehensive relevance of each candidate document is calculated based on its document relevance, initial relevance, and quality relevance.

8. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images