Literature recommendation method and related equipment

Abstract

The embodiment of the invention provides a literature recommendation method and related equipment, and is applied to the field of data processing. And carrying out P times of processing on user retrieval keywords and literature questions concerned by the user by utilizing the M large language models so as to eliminate abnormal recommendation keywords caused by illusion of the large language models. The literature retrieval platform can accurately obtain the target literature which the user is interested in through K recommendation keywords which can accurately reflect the user is interested in and by means of semantic similarity calculation, the target literature is recommended to the user to expand the view of the user, and the user is prevented from falling into an'information island '. And compared with a recommendation method based on a user portrait, the method does not need to analyze literature retrieval and reading behavior data of a large number of users, so that literature can be accurately recommended to the users, and the universality of literature recommendation is enhanced.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a document recommendation method and related equipment. Background Art

[0002] Literature recommendation technology is essentially information filtering technology, which aims to infer users' interest preferences by analyzing their literature retrieval and browsing behaviors, and recommend other literature of interest to users based on the behavioral data of similar users, thereby improving user stickiness.

[0003] At present, there are two types of document recommendation technologies. One is a content-based recommendation method, which analyzes the characteristics of the hit documents based on the user's document search terms and recommends documents with similar content to the documents retrieved by the user. However, since the documents recommended by the user's document search terms are limited, it is difficult to broaden the user's document reading horizons, and it is easy for the user to fall into an "information island". The other is a recommendation method based on user portraits, which uses the fact that users with the same or similar portraits have similar interests and preferences to recommend documents to users. This method requires analyzing a large amount of user behavior data and obtaining accurate user images in order to accurately recommend documents to users. This method has poor universality and is not suitable for scenarios where a large amount of user behavior data is lacking.

[0004] Therefore, a literature recommendation method is provided. Compared with the content-based recommendation method, it can broaden the user's literature reading horizon and prevent the user from falling into an "information island". Compared with the recommendation method based on user portraits, it can accurately recommend literature to users and has strong universality. Summary of the invention

[0005] The embodiments of the present application provide a document recommendation method and related devices for broadening the user's document reading horizons, preventing the user from falling into an "information island", and helping to accurately recommend documents to users, thereby enhancing the universality of document recommendations.

[0006] In a first aspect, an embodiment of the present application provides a document recommendation method, the method comprising: Obtaining user search keywords and concerned document titles; the user search keywords are keywords entered by the user on the document search platform, and the concerned document titles are document titles corresponding to documents that the user has browsed after clicking on the details of the documents searched from the document database by the document search platform based on the user search keywords; M large language models are used to process the user search keywords and the concerned document titles P times to obtain K recommended keywords; M is a positive integer, P is a positive integer, and the product of M and P is an integer greater than 1, and K is a positive integer, and K is less than or equal to the product of M and P; the recommended keywords are related to the user search keywords and the concerned document titles, but are different from the user search keywords; each large language model processes the user search keywords and the concerned document titles each time based on the same prompt word; According to the semantic similarity between the K recommended keywords and the document information in the document database, N target documents are determined; and the N target documents are recommended; wherein N is a positive integer.

[0007] Optionally, the user search keywords and the concerned document titles are processed P times using M large language models to obtain K recommended keywords, including: Using M large language models, the user's search keywords and the concerned document titles are processed P times to obtain P×M candidate recommended keywords; Filter out the keywords to be deleted whose semantic similarity threshold with other candidate recommended keywords is less than the first semantic similarity threshold from the P×M candidate recommended keywords; Deleting the to-be-deleted keyword from the P×M candidate recommended keywords to obtain K1 to-be-processed keywords; The K recommended keywords are obtained by merging the keywords whose semantic similarity is greater than or equal to a second semantic similarity threshold among the K1 keywords to be processed, and deleting the keywords that are the same as the user search keywords; the second semantic similarity threshold is greater than the first semantic similarity threshold.

[0008] Optionally, the method further comprises: Using the target large language model, based on the judgment prompt word, the K recommended keywords are processed to obtain the target recommended keywords that the user is interested in from the K recommended keywords; The judgment prompt word is used to judge whether each of the K recommended keywords is a keyword of interest to the user; the target large language model belongs to the M large language models; The step of determining N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database includes: According to the semantic similarity between the target recommendation keyword and the document information in the document database, N target documents are determined.

[0009] Optionally, determining N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database includes: Calculate the matching semantic similarity between each of the K recommended keywords and all document titles in the document database; For each recommended keyword, the following steps are performed: sorting all document titles in the document database according to the matching semantic similarity, and taking the documents corresponding to the document titles ranked in the top P positions of the matching semantic similarity as the target documents, where P is a positive integer.

[0010] Optionally, calculating the matching semantic similarity between each of the K recommended keywords and all document titles in the document database includes: The KL divergence algorithm is used to calculate the matching semantic similarity between each of the K recommended keywords and all the document titles in the document database.

[0011] Optionally, obtaining the title of the concerned document includes: Analyze user behavior data recorded in logs; The document titles corresponding to the documents whose browsing time by the user is greater than a preset time threshold, or the documents whose full texts are clicked and downloaded, are taken as the concerned document titles.

[0012] Optionally, the recommending the N target documents includes: The document author, title, journal name, year, volume and issue of each target document in the N target documents are displayed on the user data page of the document retrieval platform.

[0013] Optionally, there are at least two different large language models among the M large language models.

[0014] In a second aspect, an embodiment of the present application provides a document recommendation device, the device comprising: An acquisition unit is used to acquire user search keywords and concerned document titles; the user search keywords are keywords entered by the user on the document search platform, and the concerned document titles are document titles corresponding to documents browsed by the user from documents searched from the document database by the document search platform based on the user search keywords; A processing unit, configured to use M large language models to process the user search keywords and the concerned document titles P times to obtain K recommended keywords; M is a positive integer, P is a positive integer, and the product of M and P is an integer greater than 1, and K is a positive integer, and K is less than or equal to the product of M and P; the recommended keywords are related to the user search keywords and the concerned document titles, but are different from the user search keywords; each large language model processes the user search keywords and the concerned document titles each time based on the same prompt word; The recommendation unit is used to determine N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database; and recommend the N target documents.

[0015] In a third aspect, an embodiment of the present application provides a computer storage medium for storing a computer program, which, when executed, is used to implement the document recommendation method as described in any one of the first aspects.

[0016] In a fourth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on at least one computing device, enables the at least one computing device to implement the document recommendation method as described in any one of the first aspects.

[0017] Beneficial Effects

[0018] The embodiment of the present application provides a document recommendation method and related equipment. When executing the method: first obtain the user search keywords entered by the user in the document server system, and the document titles of interest that the user browses and searches from the document database based on the user search keywords, so as to fully reflect the user's intention. Then, use M large language models to process the user search keywords and the document titles of interest P times to obtain K recommended keywords, so as to use the natural language understanding and generation capabilities of the large language model to mine the user search keywords and the document titles of interest. Multiple recommended keywords that are of interest to the user and valuable are hidden in the user search keywords and the document titles of interest. Among them, the recommended keywords are related to the user search keywords and the document titles of interest, but different from the user search keywords; each large language model processes the user search keywords and the document titles of interest once with the same prompt words, so as to avoid the model output deviation caused by different prompt words, and further improve the accuracy of the generated recommended keywords. Considering that the large language model may have hallucination problems, resulting in low accuracy of the generated recommended keywords. The embodiment of the present application can use multiple large language models to process the user search keywords and the document titles of interest multiple times to eliminate abnormal recommended keywords caused by the hallucination of the large language model. Through multiple recommended keywords that can accurately reflect the user's interests and using semantic similarity calculation, the user can accurately obtain the target document that the user is interested in, and recommend the target document to the user to expand the user's horizons. In summary, the embodiment of the present application uses M large language models to process the user's search keywords and the title of the document of interest P times to obtain K recommended keywords, and performs semantic similarity calculation based on the K recommended keywords to obtain the target document, which can broaden the user's literature reading horizons and prevent the user from falling into an "information island". Compared with the recommendation method based on the user screen, it does not need to analyze the document retrieval and reading behavior data of a large number of users, so it helps to accurately recommend documents to users and enhance the universality of document recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is an application scenario diagram of a literature recommendation technology; Figure 2 A flow chart of a document recommendation method provided in an embodiment of the present application; Figure 3 A flow chart of another document recommendation method provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of a document recommendation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0022] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0023] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0024] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0025] As mentioned above, document recommendation technology is essentially information filtering technology, which aims to improve user stickiness by analyzing users' document retrieval and browsing behaviors, inferring users' interests and preferences, and recommending other documents of interest to users based on the behavioral data of similar users.

[0026] Attached Figure 1 FIG. 1 is an application scenario diagram of a document recommendation technology. The document recommendation technology is applied to a document retrieval platform 10 (also referred to as a document retrieval platform). The document retrieval platform 10 includes a terminal device 101 with a display function, a server 103 providing a processing function, and a network 102. The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0027] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. The terminal device 101 provides an interactive interface, and the user can enter the user search keyword based on the interactive interface, and send the user search keyword entered by the user to the server 103. In the embodiment of the present application, the terminal device 101 can be a user terminal such as a mobile phone, a tablet computer, and a notebook computer for document retrieval services.

[0028] The server 103 may be a server that provides various services, such as a background management server that provides support for the terminal device 101 (for example only). The server 103 performs a document search based on the user's search keyword, and sends the matched document to the terminal device 101 to be displayed on the interactive interface, thereby recommending it to the user. In one example, the server 103 has a built-in document database that stores a large number of documents. The server 103 matches documents from the document database based on the user's search keyword and recommends them to the user.

[0029] It should be noted that the document recommendation method provided in the embodiment of the present invention is generally executed by the server 103, and accordingly, the computing device is generally arranged in the server 103. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to actual needs, there may be any number of terminal devices, networks and servers.

[0030] At present, there are two types of document recommendation methods. One is a recommendation method based on document content, which further analyzes the characteristics of the hit documents, such as keywords, subject terms, classification labels, authors and journal names, based on the user's search keywords, and recommends documents similar to the user's search keywords. For example, documents with the same keywords, subject terms, classification labels, authors or journal names as the search keywords are recommended. However, the content of the documents recommended in this way is relatively limited, and the implicit semantic relationship between documents cannot be captured. It can only be based on displayed vocabulary matching, which makes it difficult to broaden the user's literature reading horizons and trap users in "information islands".

[0031] Another method is the recommendation method based on user portraits, which is based on the interactive behavior data between users and the literature retrieval platform. It is believed that users with the same or similar portraits usually have similar research directions and literature reading preferences, and thus recommends browsing literature to users with the same or similar portraits. However, this method requires analyzing the literature retrieval and reading behavior data of a large number of users, and is not suitable for platforms that lack a large amount of user behavior data.

[0032] In view of this, the embodiment of the present application provides a document recommendation method, which processes the user's search keywords and the document titles of interest multiple times through multiple large language models to eliminate abnormal recommended keywords caused by the illusion of the large language model. Through multiple recommended keywords that can accurately reflect the user's interests and use semantic similarity calculation, the user can accurately obtain the target document of interest to the user, and recommend the target document to the user to expand the user's horizons. In summary, the embodiment of the present application uses M large language models to process the user's search keywords and the document titles of interest P times to obtain K recommended keywords, and performs semantic similarity calculation based on the K recommended keywords to obtain the target document, which can broaden the user's literature reading horizons and prevent the user from falling into an "information island". Compared with the recommendation method based on the user screen, it does not need to analyze the document retrieval and reading behavior data of a large number of users, so it helps to accurately recommend documents to users and enhance the universality of document recommendations.

[0033] The following describes in detail the document recommendation method provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0034] Attached Figure 2 A flow chart of a document recommendation method provided in an embodiment of the present application, the method is applied on a document retrieval platform 10, and the method includes the following contents: S21. Obtain the user's search keywords and the titles of the documents of interest.

[0035] The user search keywords are keywords input by the user in the interactive interface of the terminal device 101 of the document search platform 10. The user search keywords provided in the embodiment of the present application can be one or more, and are not specifically limited. For example, the user search keywords are "artificial intelligence", "image recognition" and "medical imaging". The user search keywords can directly reflect the user's initial query intention.

[0036] The document titles of interest are the document titles corresponding to the documents that the user has clicked on the details page and browsed, among the documents that the document retrieval platform 10 has searched from the document database based on the user's search keywords. The document titles of interest can be one or more, and the embodiments of the present application are not specifically limited. For example, the document titles of interest are "Application of Artificial Intelligence in Medical Image Analysis" and "Application of Deep Learning Algorithms in Medical Imaging". It can be understood that the document titles of interest are closer to the actual needs of users.

[0037] The document title of interest may be a document title corresponding to a document whose browsing time by the user is greater than or equal to a preset time threshold, or a document title corresponding to a document whose full text the user clicks, or a document title corresponding to a document whose full text the user downloads, etc., which is not specifically limited in the embodiments of the present application. The preset time threshold is a time length set by a person skilled in the art as required, such as 1 minute.

[0038] It can be understood that the user's search keywords and the literature titles they are interested in can more fully reflect the user's query intentions compared to a single information source.

[0039] In an embodiment of the present application, the document retrieval platform can analyze the user behavior data recorded in the log, obtain the user's user search keywords, and use the document titles corresponding to the documents whose browsing time is greater than or equal to a preset time threshold, the user clicks on the full text, and the user downloads the full text as the document titles of interest.

[0040] S22. Use M large language models to process the user's search keywords and the titles of the documents he / she is interested in P times to obtain K recommended keywords.

[0041] Wherein, M is a positive integer, and P is a positive integer, but the value of M×P is an integer greater than 1.

[0042] Recommended keywords refer to the recommended words related to the user's search keywords and the titles of the documents that the user is interested in, and are used to reflect the implicit semantics in the user's search keywords and the titles of the documents that the user is interested in. Recommended keywords are used to expand the user's query scope and help users find documents in more research fields, thereby avoiding "information islands". In the embodiment of the present application, the recommended keywords are different from the user's search keywords, so as to provide more search opportunities and discover related documents that were not originally considered.

[0043] The embodiment of the present application utilizes a large language model to process user search keywords and document titles of interest. The language understanding capability of the large language model can be utilized to mine the semantic information implicit in the user search keywords and document titles of interest, and recommend keywords of interest to the user.

[0044] However, large language models may cause hallucination problems, resulting in the acquisition of recommended keywords that do not conform to the user's query intent. In view of this, an embodiment of the present application proposes using M large language models to process user search keywords and document titles of interest P times to obtain K recommended keywords. For example, one large language model is used to process user search keywords and document titles of interest multiple times to obtain K recommended keywords, or multiple large language models are used to process user search keywords and document titles of interest at least once to obtain K recommended keywords. Thus, by using M large language models to process user search keywords and document titles of interest P times, the hallucination problem caused by word processing by a single large language model is reduced, and the accuracy of the acquired recommended keywords is improved. It can be understood that the higher the accuracy of the recommended keywords, the more accurately they reflect the user's query intent.

[0045] It is understandable that the larger the value of M, the higher the accuracy of the recommended keywords. However, a large language model requires computing resources. If the large language model is too large, it will occupy a lot of computing resources, resulting in slow computing speed, and the layout of multiple large language models will also increase deployment difficulties. Therefore, M should not be too large. In one example, the value of M can be 3.

[0046] Among them, each time each large language model processes the user's search keywords and the title of the document of interest, in order to make the recommended keyword results obtained more accurate, the document retrieval platform 10 uses the same prompt words. Prompt words are an important tool for guiding the large language model to generate specific types of outputs. By using the same prompt words, the system can more accurately control the output direction of the model and ensure that the generated recommended keywords meet the user's query intentions and needs. By using the same prompt words, the system can better guide the model to generate reasonable recommended keywords and reduce the occurrence of hallucination problems.

[0047] Furthermore, after obtaining the recommended keywords, the target large language model can be used to process the K recommended keywords based on the judgment prompt words, and obtain the target recommended keywords that the user is interested in from the K recommended keywords. Among them, the target large language model belongs to the M large language models.

[0048] The judgment prompt word is used to judge whether each of the K recommended keywords is a keyword that the user is interested in. For example, the judgment prompt word is "whether the above recommended keywords are keywords that the user is interested in". By using the multi-round question-answering capability of the large language model, the target recommended keywords that the user is more interested in and can better reflect the user's query intention can be further obtained.

[0049] In actual use, one of the M large language models can be designated as the target large language model, or a large language model can be randomly assigned as the target large language model based on the M large language models, which is not specifically limited in the embodiments of the present application.

[0050] S23. Determine N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database.

[0051] The document information includes document title, document abstract, document keywords, etc. For better explanation, the document title is used as an example for schematic explanation below.

[0052] The target document refers to a document in the database that is highly similar to the recommended keyword. In one example, the target document may be a document whose semantic similarity to the recommended keyword is greater than a preset semantic similarity threshold T. Where 0≤T≤1; the larger T is, the more interested the user is in the recommended document, for example, T=0.8. In another example, the target document is a document that ranks in the top P positions in semantic similarity to each recommended keyword. P is a positive integer, for example, P=2.

[0053] In one example, the document retrieval platform 10 calculates the semantic similarity between each of the K recommended keywords and the document information (eg, document title) in the document database, and selects documents having a semantic similarity greater than or equal to T, which are target documents.

[0054] For example, the recommended keywords generated by the large language model include 3, namely, recommended keyword 1: medical image segmentation; recommended keyword 2: application of transfer learning in medical images; recommended keyword 3: application of weakly supervised learning in medical images. The literature retrieval platform 10 calculates the semantic similarity between each of the 3 recommended keywords and the title of the document in the literature database, for example, the following semantic similarity result table is obtained, as shown in Table 1.

[0055] Table 1

[0056] If T is 0.8, the target documents include documents corresponding to Topic 1, Topic 3 and Topic 4.

[0057] In another example, the document retrieval platform 10 obtains the documents with the top P semantic similarity in each recommended keyword as target documents. For example, P=1, as shown in Table 1, the target document corresponding to the recommended keyword 1 is the document corresponding to the title 3, the target document corresponding to the recommended keyword 2 is the document corresponding to the title 4, and the target document corresponding to the recommended keyword 3 is the document corresponding to the title 4.

[0058] It should be noted that in the embodiment of the present application, for duplicate target documents, duplicates are removed and merged before recommendation. For example, the target document corresponding to the recommended keyword 2 and the target document corresponding to the recommended keyword 3 are both documents corresponding to topic 4. The two target documents are removed, that is, a document corresponding to topic 4 is recommended to the user.

[0059] It should be noted that in the embodiment of the present application, the document retrieval platform 10 can use the KL algorithm to calculate the semantic similarity between each recommended keyword and the document title in the document database. The KL divergence (KL-divergePce) algorithm, also known as the Kullback-Leibler divergePce, is an indicator used to measure the difference between two probability distributions.

[0060] In an embodiment of the present application, if the recommended keyword is a target recommended keyword, the embodiment of the present application calculates the semantic similarity between the target recommended keyword and the document information in the document database, determines N target documents, thereby further improving the target document to better reflect the user's query intention.

[0061] S24. Recommend N target documents.

[0062] For example, the document retrieval platform 10 displays the author, title, journal name, year, volume and issue of the target document in a reference format on the user data page of the terminal device 101, thereby intelligently recommending the target document to the user in a personalized manner.

[0063] In summary, the document recommendation method provided in the embodiment of the present application first obtains the user search keywords entered by the user on the document search platform, and the document titles of interest searched from the document database based on the user search keywords browsed by the user, and fully reflects the user's intention through contextual learning. Then, M large language models are used to process the user search keywords and the document titles of interest P times to obtain K recommended keywords, so as to utilize the natural language understanding and generation capabilities of the large language model to mine the user's interest and valuable multiple recommended keywords implied in the user search keywords and the document titles of interest. Among them, the recommended keywords are related to the user search keywords and the document titles of interest, but different from the user search keywords; each large language model processes the user search keywords and the document titles of interest once with the same prompt words, so as to avoid the model output deviation caused by different prompt words, and further improve the accuracy of the generated recommended keywords. Considering that the large language model may have hallucination problems, resulting in low accuracy of the generated recommended keywords. The embodiment of the present application can use multiple large language models to process the user search keywords and the document titles of interest multiple times to eliminate abnormal recommended keywords caused by the hallucination of the large language model. Through multiple recommended keywords that can accurately reflect the user's interests and the use of semantic similarity calculation, the user can accurately obtain the target document that the user is interested in, and recommend the target document to the user to expand the user's horizons. In summary, the embodiment of the present application uses M large language models to process the user's search keywords and the title of the document of interest P times to obtain K recommended keywords, and performs semantic similarity calculation based on the K recommended keywords to obtain the target document, which can broaden the user's literature reading horizons and prevent the user from falling into an "information island". And compared with the recommendation method based on the user screen, it is not necessary to analyze the document retrieval and reading behavior data of a large number of users, so it is helpful to accurately recommend documents to users and enhance the universality of document recommendations. Furthermore, due to different users, the user search keywords and the title of the document of interest entered by the user are different, so the recommended documents provided by the embodiment of the present application are also personalized and customized.

[0064] The following takes M large language models to process user search keywords and document titles of interest as an example to further illustrate the document recommendation method provided in the embodiment of the present application.

[0065] Attached Figure 3 Another method flow chart of document recommendation provided in the present application embodiment, the method is applied to the attached Figure 1 The document retrieval platform 10 shown in the figure comprises the following contents: ①: Initial data collection.

[0066] In the embodiment of the present application, the document search platform 10 analyzes the user behavior, finds the hit documents based on the user's keywords, and further pays attention to the user's browsing and the focused document titles as the focused document titles. Among them, the focused document titles are the document titles corresponding to the documents whose browsing time by the user is greater than or equal to the preset time threshold, and the user clicks on the full text, and the user downloads the full text. It can be obtained by analyzing the user behavior data recorded in the log.

[0067] The initial data include the user's error detection keywords and the titles of the documents he / she is interested in.

[0068] ②: Large language model generates personalized recommendation literature keywords.

[0069] Among them, the personalized recommended literature keywords are the recommended keywords.

[0070] The user's user search keywords and the document titles that the user is interested in are a scenario reproduction of the documents and research directions that the user is interested in. Using the idea of ​​scenario learning, the user's user search keywords and the document titles that the user is interested in are used as initial data and sent to the large language model. In the embodiment of the present application, M large language models are used, namely large language model 1, large language model 2, ..., large language model P, to process the user's search keywords and document titles that the user is interested in, respectively. For example, if the prompt word clearly requires the number to be 1, then large language model 1 obtains recommended document keyword 1, large language model 2 obtains recommended document keyword 2, ... large language model M obtains recommended document keyword M.

[0071] Each large language model uses the same prompt word for processing.

[0072] To avoid the hallucination problem of large language models, it can be further processed in the following ways: Step 1: Use M large language models to process the user's search keywords and the document titles of interest P times to obtain P×M candidate recommended keywords.

[0073] In the embodiment of the present application, M recommended document keywords are obtained.

[0074] Step 2: Filter out the keywords to be deleted whose semantic similarity with other candidate recommended keywords is less than a first semantic similarity threshold from the P×M candidate recommended keywords.

[0075] Step 3: Delete the keywords to be deleted from the P×M initial recommended keywords to obtain K1 keywords to be processed.

[0076] Wherein, K1 is a positive integer, and K1 is less than or equal to P×M.

[0077] It is understandable that if a large language model has hallucinations, the recommended document keywords generated by the large language model will be greatly different from those generated by other language models, and the obtained semantic similarity will be small. Therefore, the keywords with semantic similarity less than the first semantic similarity threshold are selected from the M recommended document keywords, and the keywords are marked as keywords to be deleted. Therefore, the recommended document keywords generated by the hallucination of the large language model can be extracted from the M recommended document keywords, and K1 document keywords to be processed are obtained.

[0078] The first semantic similarity threshold is a preset parameter used to control the semantic similarity between keywords. Usually, this threshold is set relatively low (e.g., 0.2 or lower) to ensure that the retained keywords do not contain keywords generated by large language model hallucination, thereby improving the accuracy of recommendation.

[0079] Step 4: Merge the keywords whose semantic similarity is greater than or equal to the second semantic similarity threshold among the K1 keywords to be processed, and delete the keywords that are the same as the user's search keyword to obtain k recommended keywords.

[0080] Wherein, k is a positive integer and k≤K1.

[0081] It can be understood that by removing duplicate keywords and eliminating keywords that are the same as the user's search keywords, K recommended keywords are obtained to obtain a recommended document keyword list, ensuring that the recommended keywords are diverse and maintain a certain difference from the user's initial query, thereby helping users discover more valuable documents.

[0082] The second semantic similarity threshold is usually lower than the first semantic similarity threshold and is used to determine which keywords have sufficiently high semantic similarity to be merged. For example, if the semantic similarity of two keywords is greater than or equal to 0.7, they are considered to be highly similar and can be merged into one keyword.

[0083] ③: KL-divergePce semantic similarity calculation.

[0084] The document retrieval platform 10 uses the KL-divergePce algorithm to calculate the vector semantic similarity between each keyword in the recommended document keyword list and the document title in the document database.

[0085] Furthermore, the document titles are sorted according to the semantic similarity from large to small, or from small to large.

[0086] ④: Personalized intelligent recommendation of literature.

[0087] In one example, N documents with semantic similarity greater than a preset semantic similarity threshold T are selected from the largest to the smallest semantic similarity and recommended to the user, wherein N is a value preset by a person skilled in the art as required, and N is a positive integer, for example, N=20.

[0088] Specifically, the selection method can be: the document retrieval platform 10 first selects N1 documents whose semantic similarity is greater than a preset semantic similarity threshold T. If N1>N, then delete the documents with the lower semantic similarity ranking to obtain N documents recommended to the user. If N1=N, then the N documents are directly recommended to the user. If N1<N, then for the recommended document keywords that the k users are interested in, select the top (N-N1) documents with semantic similarity less than or equal to T, and recommend the total N documents obtained to the user.

[0089] In another example, in the embodiment of the present application, in step ②, there are k recommended document keywords that the user is interested in. For each group of semantic similarities between the recommended document keywords and the document title, the first |N / K| documents are selected.

[0090] Among them, |N / K| means that the result of N / K is rounded to the nearest integer.

[0091] If the total number of documents selected here exceeds or falls short of N due to rounding, the corresponding documents will be discarded or added according to the semantic similarity, so that the total number of personalized recommended documents is N.

[0092] Specifically, the document retrieval platform 10 obtains information such as the author, title, journal name, year and volume of the document from the document database, and recommends the above information in a reference format to the corresponding user on the user data page of the terminal device 101 in a targeted and intelligent manner.

[0093] It should be noted that the embodiment of the present application limits the number N recommended to the user to avoid causing trouble to the user due to too much or too little information, and helps the user to obtain more valuable information without wasting documents due to too many documents.

[0094] In summary, the above method uses the idea of ​​situational learning to allow the large language model to automatically generate literature keywords that users are interested in. Calling multiple large language models can not only avoid the risk of incorrect results caused by the hallucination problem of a single large language model, but also accurately generate multiple literature keywords that users are interested in to expand the user's vision. Ultimately, personalized intelligent recommended literature meets user expectations and is favored by users, thereby improving the user stickiness of the literature retrieval platform and increasing the number of visits to the literature retrieval platform.

[0095] In addition, an embodiment of the present application also provides a document recommendation device.

[0096] Attached Figure 4 The schematic diagram of the structure of the document recommendation device provided in the embodiment of the present application is as follows: The acquisition unit 401 is used to acquire the user's search keyword and the document title of interest; the user's search keyword is the keyword entered by the user on the document search platform, and the document title of interest is the document title corresponding to the document browsed by the user from the document database searched by the document search platform based on the user's search keyword; The processing unit 402 is used to use M large language models to process the user's search keywords and the document title of interest P times to obtain K recommended keywords; M is a positive integer, P is a positive integer, and the product of M and P is an integer greater than 1, and K is a positive integer, and K is less than or equal to the product of M and P; the recommended keywords are related to the user's search keywords and the document title of interest, but are different from the user's search keywords; each large language model processes the user's search keywords and the document title of interest once and has the same prompt words; The recommendation unit 403 is used to determine N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database; and recommend the N target documents.

[0097] The processing unit 402 is specifically used for: Use M large language models to process the user's search keywords and the titles of the literature that the user is interested in P times to obtain P×M candidate recommended keywords; filter out the keywords to be deleted from the P×M candidate recommended keywords whose semantic similarity threshold with other candidate recommended keywords is less than the first semantic similarity threshold; delete the keywords to be deleted from the P×M initial recommended keywords to obtain K1 keywords to be processed; merge the keywords whose semantic similarity in the K1 keywords to be processed is greater than or equal to the second semantic similarity threshold, and delete the keywords that are the same as the user's search keywords to obtain k recommended keywords; the second semantic similarity threshold is greater than the first semantic similarity threshold.

[0098] Optionally, the recommendation unit 403 is specifically used to: calculate the matching semantic similarity between each recommended keyword among the K recommended keywords and all document titles in the document database; execute for each recommended keyword: sort all document titles in the document database according to the matching semantic similarity, and take the documents corresponding to the document titles ranked in the top P places in the matching semantic similarity as the target documents, where P is a positive integer.

[0099] Optionally, calculating the matching semantic similarity between each of the K recommended keywords and all document titles in the document database includes: calculating the matching semantic similarity between each of the K recommended keywords and all document titles in the document database using a KL divergence algorithm.

[0100] Optionally, the acquisition unit 401 is specifically used to: analyze the user behavior data recorded in the log; and take the document titles corresponding to the documents whose browsing time by the user is greater than a preset time threshold, or the documents whose full text is clicked and downloaded, as the document titles of interest.

[0101] Optionally, the recommendation unit 403 is further configured to display the document author, title, journal name, year, volume and issue of each target document among the N target documents on a user data page of the document retrieval platform.

[0102] Among them, there are at least 2 unused large language models among the M large language models.

[0103] In summary, the document recommendation device provided in the embodiment of the present application can use multiple large language models to process the user's search keywords and the document titles of interest multiple times to eliminate abnormal recommended keywords caused by the illusion of the large language model. Through multiple recommended keywords that can accurately reflect the user's interests and use semantic similarity calculation, the user can accurately obtain the target document of interest to the user, and recommend the target document to the user to expand the user's horizons. In summary, the embodiment of the present application uses M large language models to process the user's search keywords and the document titles of interest P times to obtain K recommended keywords, and performs semantic similarity calculation based on the K recommended keywords. The target document obtained can broaden the user's literature reading horizons and prevent the user from falling into an "information island". Compared with the recommendation method based on the user screen, it does not need to analyze the document retrieval and reading behavior data of a large number of users, so it helps to accurately recommend documents to users and enhance the universality of document recommendations.

[0104] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device, the computing device is caused to execute the above-mentioned document recommendation method. The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc. The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned document recommendation method.

[0105] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0106] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A document recommendation method, characterized in that: The method comprises: Obtaining user search keywords and concerned document titles; the user search keywords are keywords entered by the user on the document search platform, and the concerned document titles are document titles corresponding to documents that the user has browsed after clicking on the details of the documents searched from the document database by the document search platform based on the user search keywords; M large language models are used to process the user search keywords and the concerned document titles P times to obtain K recommended keywords; M is a positive integer, P is a positive integer, and the product of M and P is an integer greater than 1, and K is a positive integer, and K is less than or equal to the product of M and P; the recommended keywords are related to the user search keywords and the concerned document titles, but are different from the user search keywords; each large language model processes the user search keywords and the concerned document titles each time based on the same prompt word; According to the semantic similarity between the K recommended keywords and the document information in the document database, N target documents are determined; and the N target documents are recommended; wherein N is a positive integer.

2. The method according to claim 1, characterized in that The M large language models are used to process the user's search keywords and the concerned document titles P times to obtain K recommended keywords, including: Using M large language models, the user's search keywords and the concerned document titles are processed P times to obtain P×M candidate recommended keywords; Filter out the keywords to be deleted whose semantic similarity threshold with other candidate recommended keywords is less than the first semantic similarity threshold from the P×M candidate recommended keywords; Deleting the to-be-deleted keyword from the P×M candidate recommended keywords to obtain K1 to-be-processed keywords; The K recommended keywords are obtained by merging the keywords whose semantic similarity is greater than or equal to a second semantic similarity threshold among the K1 keywords to be processed, and deleting the keywords that are the same as the user search keywords; the second semantic similarity threshold is greater than the first semantic similarity threshold.

3. The method according to claim 1, characterized in that The method further comprises: Using the target large language model, based on the judgment prompt word, the K recommended keywords are processed to obtain the target recommended keywords that the user is interested in from the K recommended keywords; The judgment prompt word is used to judge whether each of the K recommended keywords is a keyword of interest to the user; the target large language model belongs to the M large language models; The step of determining N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database includes: According to the semantic similarity between the target recommendation keyword and the document information in the document database, N target documents are determined.

4. The method according to claim 1, characterized in that The step of determining N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database includes: Calculate the matching semantic similarity between each of the K recommended keywords and all document titles in the document database; For each recommended keyword, the following steps are performed: sorting all document titles in the document database according to the matching semantic similarity, and taking the documents corresponding to the document titles ranked in the top P positions of the matching semantic similarity as the target documents, where P is a positive integer.

5. The method according to claim 4, characterized in that The calculating of the matching semantic similarity between each of the K recommended keywords and all the document titles in the document database includes: The KL divergence algorithm is used to calculate the matching semantic similarity between each of the K recommended keywords and all the document titles in the document database.

6. The method according to claim 4, characterized in that Get the title of the document of interest, including: Analyze user behavior data recorded in logs; The document titles corresponding to the documents whose browsing time by the user is greater than a preset time threshold, or the documents whose full texts are clicked and downloaded, are taken as the concerned document titles.

7. The method according to claim 1, characterized in that The recommending of the N target documents includes: The document author, title, journal name and / or year, volume and issue of each target document in the N target documents are displayed on the user data page of the document retrieval platform.

8. The method according to claim 1, characterized in that: There are at least two different large language models among the M large language models.

9. A document recommendation device, characterized in that: The device comprises: An acquisition unit is used to acquire user search keywords and concerned document titles; the user search keywords are keywords entered by the user on the document search platform, and the concerned document titles are document titles corresponding to documents browsed by the user from documents searched from the document database by the document search platform based on the user search keywords; A processing unit, configured to use M large language models to process the user search keywords and the concerned document titles P times to obtain K recommended keywords; M is a positive integer, P is a positive integer, and the product of M and P is an integer greater than 1, and K is a positive integer, and K is less than or equal to the product of M and P; the recommended keywords are related to the user search keywords and the concerned document titles, but are different from the user search keywords; each large language model processes the user search keywords and the concerned document titles each time based on the same prompt word; The recommendation unit is used to determine N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database; and recommend the N target documents.

10. A computer storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the document recommendation method according to any one of claims 1 to 8.

Images