A literature recommendation method and related devices

Abstract

An embodiment of the present application provides a literature recommendation method and related devices, which are applied to the field of data processing. Using M large language models, the user's retrieval keywords and the titles of the concerned literature are processed P times to eliminate abnormal recommended keywords caused by large language model hallucinations. The literature retrieval platform can accurately obtain the target literature of interest to the user through K recommended keywords that can accurately reflect the user's interests, and use semantic similarity calculation to recommend the target literature to the user to expand the user's horizons and prevent the user from falling into an "information island". Moreover, compared with the recommendation method based on user portraits, it is not necessary to analyze a large amount of literature retrieval and reading behavior data of users. Therefore, it helps to accurately recommend literature for users and enhance the universality of literature recommendation.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular, to a literature recommendation method and related devices. Background Art

[0002] The literature recommendation technology is essentially an information filtering technology, aiming to infer the user's interest preferences by analyzing the user's literature retrieval and browsing behaviors, and combining the behavior data of similar users to recommend other literature that the user is interested in, thereby improving user stickiness.

[0003] Currently, there are two types of literature recommendation technologies. One is the content-based recommendation method, that is, based on the user's file retrieval terms, analyzing the characteristics of the retrieved literature, and recommending literature similar to the content of the user's retrieved literature. However, due to the limited literature recommended by the user's file retrieval terms, it is difficult to broaden the user's literature reading horizon and easily makes the user fall into an "information island". The other is the user portrait-based recommendation method, which uses the fact that users with the same or similar portraits have similar interest preferences to recommend literature to users. This method requires analyzing the behavior data of a large number of users to obtain accurate user portraits in order to accurately recommend literature to users. This method has poor universality and is not applicable to scenarios lacking a large amount of user behavior data.

[0004] Therefore, providing a literature recommendation method can broaden the user's literature reading horizon compared to the content-based recommendation method, avoid the user falling into an "information island", and can accurately recommend literature to users and has strong universality compared to the user portrait-based recommendation method. Summary of the Invention

[0005] Embodiments of this application provide a literature recommendation method and related devices, which are used to broaden the user's literature reading horizon, avoid the user falling into an "information island", and help to accurately recommend literature to users and enhance the universality of literature recommendation.

[0006] In a first aspect, embodiments of this application provide a literature recommendation method, and the method includes:

[0007] Obtain the user's retrieval keywords and the titles of the concerned literature; the user's retrieval keywords are the keywords input by the user on the literature retrieval platform, and the titles of the concerned literature are the titles of the literature corresponding to the literature that the user has browsed after clicking on the details among the literatures searched from the literature database by the literature retrieval platform based on the user's retrieval keywords;

[0008] Use M large language models to process the user's retrieved keywords and the titles of the concerned documents P times to obtain K recommended keywords; M is a positive integer, P is a positive integer, and the product of M and P is an integer greater than 1, K is a positive integer, and K is less than or equal to the product of M and P; the recommended keywords are related to the user's retrieved keywords and the titles of the concerned documents, but are different from the user's retrieved keywords; each large language model processes the user's retrieved keywords and the titles of the concerned documents each time based on the same prompt words;

[0009] Determine N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database; recommend the N target documents; N is a positive integer.

[0010] Optionally, the use of M large language models to process the user's retrieved keywords and the titles of the concerned documents P times to obtain K recommended keywords includes:

[0011] Use M large language models to process the user's retrieved keywords and the titles of the concerned documents P times to obtain P×M candidate recommended keywords;

[0012] Screen out the keywords to be deleted whose semantic similarity threshold with other candidate recommended keywords is less than the first semantic similarity threshold from the P×M candidate recommended keywords;

[0013] Delete the keywords to be deleted from the P×M candidate recommended keywords to obtain K1 keywords to be processed;

[0014] Merge the keywords whose semantic similarity is greater than or equal to the second semantic similarity threshold among the K1 keywords to be processed, and delete the keywords that are the same as the user's retrieved keywords to obtain the K recommended keywords; the second semantic similarity threshold is greater than the first semantic similarity threshold.

[0015] Optionally, the method further includes:

[0016] Use a target large language model to process the K recommended keywords based on a judgment prompt word to obtain target recommended keywords that the user is interested in from the K recommended keywords;

[0017] Among them, the judgment prompt word is used to judge whether each recommended keyword among the K recommended keywords is a keyword that the user is interested in; the target large language model belongs to the M large language models;

[0018] The determination of N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database includes:

[0019] Determine N target documents according to the semantic similarity between the recommended keywords for the target and the document information in the literature database.

[0020] Optionally, the determining N target documents according to the semantic similarity between the K recommended keywords and the document information in the literature database includes:

[0021] Calculate the matching semantic similarity between each of the K recommended keywords and all the document titles in the literature database;

[0022] For each of the recommended keywords, perform the following: sort all the document titles in the literature database according to the matching semantic similarity, and use the documents corresponding to the document titles ranked in the top P positions in terms of the matching semantic similarity as the target documents, where P is a positive integer.

[0023] Optionally, the calculating the matching semantic similarity between each of the K recommended keywords and all the document titles in the literature database includes:

[0024] Use the KL divergence algorithm to calculate the matching semantic similarity between each of the K recommended keywords and all the document titles in the literature database.

[0025] Optionally, obtaining the concerned document titles includes:

[0026] Analyze the user behavior data recorded in the log;

[0027] Use the document titles corresponding to the documents with the browsing time of the user greater than the preset time threshold, or the documents for which the full text is clicked and downloaded as the concerned document titles.

[0028] Optionally, the recommending the N target documents includes:

[0029] Display the document author, title, journal name, and year / volume / issue of each of the N target documents on the user data page of the literature retrieval platform.

[0030] Optionally, there are at least two different large language models among the M large language models.

[0031] In a second aspect, an embodiment of the present application provides a document recommendation device, and the device includes:

[0032] An acquisition unit for acquiring the user's retrieval keywords and the titles of the literatures of interest; the user's retrieval keywords are the keywords input by the user on the literature retrieval platform, and the titles of the literatures of interest are the titles of the literatures corresponding to the literatures browsed by the user among the literatures searched from the literature database by the literature retrieval platform based on the user's retrieval keywords;

[0033] A processing unit for performing P times of processing on the user's retrieval keywords and the titles of the literatures of interest by using M large language models to obtain K recommended keywords; M is a positive integer, P is a positive integer, and the product of M and P is an integer greater than 1, K is a positive integer, and K is less than or equal to the product of M and P; the recommended keywords are related to the user's retrieval keywords and the titles of the literatures of interest, but are different from the user's retrieval keywords; each large language model performs each processing on the user's retrieval keywords and the titles of the literatures of interest based on the same prompt;

[0034] A recommendation unit for determining N target literatures according to the semantic similarity between the K recommended keywords and the literature information in the literature database; recommending the N target literatures.

[0035] In a third aspect, an embodiment of the present application provides a computer storage medium for storing a computer program, which when executed, is used to implement the literature recommendation method according to any one of the first aspects.

[0036] In a fourth aspect, an embodiment of the present application provides a computer program product containing instructions, which when running on at least one computing device, enables at least one computing device to implement the literature recommendation method according to any one of the first aspects.

[0037] Beneficial effects

[0038] The present embodiment provides a document recommendation method and related devices. When executing the method, the following steps are performed: first, user search keywords entered by a user into a document server system, as well as the user-viewed document titles searched from a document database based on the user search keywords, are obtained to fully reflect the user's intent. Then, M large language models are used to process the user search keywords and the document titles P times to obtain K recommended keywords. Leveraging the natural language understanding and generation capabilities of the large language models, the large language models are used to mine multiple, interesting, and valuable recommended keywords implicit in the user search keywords and the document titles. The recommended keywords are related to the user search keywords and the document titles, but different from the user search keywords. Each large language model processes the user search keywords and the document titles once, using the same prompt word, to avoid model output deviations caused by different prompt words and further improve the accuracy of the generated recommended keywords. Considering that large language models may experience hallucinations, resulting in low accuracy in generated recommended keywords, the present embodiment utilizes multiple large language models to process the user search keywords and the document titles multiple times to eliminate abnormal recommended keywords caused by hallucinations of the large language models. Through multiple recommended keywords that accurately reflect the user's interests and using semantic similarity calculation, the user can accurately obtain the target document that the user is interested in, and recommend the target document to the user to expand the user's horizons. In summary, the embodiment of the present application uses M large language models to process the user's search keywords and the title of the document of interest P times to obtain K recommended keywords, and performs semantic similarity calculation based on the K recommended keywords to obtain target documents, which can broaden the user's literature reading horizons and prevent the user from falling into "information islands". Compared with the recommendation method based on the user screen, it does not need to analyze the document retrieval and reading behavior data of a large number of users, so it helps to accurately recommend documents to users and enhance the universality of document recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is an application scenario diagram of a literature recommendation technology;

[0040] Figure 2 A flow chart of a document recommendation method provided in an embodiment of the present application;

[0041] Figure 3 A flow chart of another document recommendation method provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the structure of the document recommendation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0044] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0045] The terms "first", "second", etc. in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, rather than to describe the specific order of the target objects.

[0046] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0047] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" refers to two or more. For example, a plurality of processing units refers to two or more processing units; a plurality of systems refers to two or more systems.

[0048] As described above, the literature recommendation technology is essentially an information filtering technology, which aims to infer the user's interest preferences by analyzing the user's literature retrieval and browsing behaviors, and combine the behavior data of similar users to recommend other literature that the user is interested in, thereby improving user stickiness.

[0049] Attached Figure 1 It is an application scenario diagram of a literature recommendation technology. This literature recommendation technology is applied to the literature retrieval platform 10 (also known as the literature retrieval platform). The literature retrieval platform 10 includes a terminal device 101 with a display function, a server 103 providing processing functions, and a network 102. The network 102 is used to provide a medium for the communication link between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0050] Users can use the terminal device 101 to interact with the server 103 via the network 102 to receive or send messages, etc. The terminal device 101 provides an interaction interface. Users can input user retrieval keywords based on this interaction interface and send the user retrieval keywords input by the users to the server 103. In the embodiments of the present application, the terminal device 101 can be a user terminal such as a mobile phone, a tablet computer, or a notebook for the literature retrieval service.

[0051] The server 103 can be a server that provides various services. For example, it can be a background management server that provides support for the terminal device 101 (only for example). The server 103 performs a literature retrieval based on the user retrieval keywords and sends the retrieved literature to the terminal device 101 for display on the interaction interface, so as to recommend it to the users. In one example, the server 103 has a built-in literature database that stores a large number of literatures. The server 103 matches literatures from the literature database based on the user retrieval keywords and recommends them to the users.

[0052] It should be noted that the literature recommendation method provided by the embodiments of the present invention is generally executed by the server 103. Correspondingly, the computing device is generally arranged in the server 103. It should be understood that Figure 1 the numbers of the terminal devices, networks, and servers in

[0053] At present, there are two types of literature recommendation methods. One is the recommendation method based on the literature content, that is, based on the user's retrieval keywords, further analyze the characteristics of the retrieved literatures, such as keywords, subject terms, classification labels, authors, and journal names, etc., and recommend literatures similar to the user's retrieval keywords. For example, recommend literatures with the same keywords, the same subject terms, the same classification labels, the same authors, or the same journal names as the retrieval keywords. However, the literature content recommended in this way is relatively limited, and the implicit semantic relationships between literatures cannot be captured. It can only be based on the matching of the displayed vocabulary, which is difficult to broaden the user's literature reading horizons and makes the user fall into an "information island".

[0054] The other is the recommendation method based on the user portrait, that is, recommend based on the interaction behavior data between the user and the literature retrieval platform. It is considered that users with the same or similar portraits usually have similar research directions and literature reading preferences, so as to recommend the browsed literatures of users with the same or similar portraits. However, this method requires analyzing a large amount of literature retrieval and reading behavior data of users and is not applicable to platforms lacking a large amount of user behavior data.

[0055] In view of this, the embodiments of the present application provide a literature recommendation method. Through multiple large language models, the user's retrieved keywords and the titles of the concerned literature are processed multiple times to eliminate abnormal recommended keywords caused by large language model hallucinations. The user can use multiple recommended keywords that can accurately reflect the user's interests, and through semantic similarity calculation, accurately obtain the target literature that the user is interested in, and recommend the target literature to the user to expand the user's horizons. To sum up, the embodiments of the present application use M large language models to process the user's retrieved keywords and the titles of the concerned literature P times to obtain K recommended keywords, and based on the K recommended keywords, perform semantic similarity calculation to obtain the target literature, which can broaden the user's literature reading horizons and prevent the user from falling into an "information island". And compared with the recommendation method based on the user's profile, it does not require analyzing a large amount of literature retrieval and reading behavior data of users, so it helps to accurately recommend literature for users and enhance the universality of literature recommendation.

[0056] The following will be a detailed description of the literature recommendation method provided by the embodiments of the present application with reference to the accompanying drawings.

[0057] Appendix Figure 2 FIG. is a flowchart of a literature recommendation method provided by an embodiment of the present application. This method is applied to a literature retrieval platform 10, and the method includes the following:

[0058] S21. Obtain the user's retrieved keywords and the titles of the concerned literature.

[0059] The user's retrieved keywords are the keywords input by the user on the interaction interface of the terminal device 101 of the literature retrieval platform 10. The user's retrieved keywords provided by the embodiments of the present application can be one or multiple, without specific limitation. For example, the user's retrieved keywords are "artificial intelligence", "image recognition", and "medical imaging". The user's retrieved keywords can directly reflect the user's preliminary query intention.

[0060] The titles of the concerned literature are the titles of the literature corresponding to the literature that the user has clicked on the detail page and browsed among the literature searched by the literature retrieval platform 10 based on the user's retrieved keywords. The titles of the concerned literature can be one or multiple, and the embodiments of the present application do not specifically limit. For example, the titles of the concerned literature are "Applications of Artificial Intelligence in Medical Image Analysis" and "Applications of Deep Learning Algorithms in Medical Imaging". It can be understood that the titles of the concerned literature are closer to the user's actual needs.

[0061] Among them, the title of the literature of interest can be the title of the literature corresponding to the literature whose browsing duration by the user is greater than or equal to the preset duration threshold, or the title of the literature corresponding to the literature whose full text is clicked by the user, or the title of the literature corresponding to the literature whose full text is downloaded by the user, etc. The embodiments of the present application do not specifically limit this. Among them, the preset duration threshold is a duration set by those skilled in the art according to needs, such as 1 min, etc.

[0062] It can be understood that the user's retrieval keywords and the titles of the literatures of interest can more fully reflect the user's query intention compared to a single information source.

[0063] In the embodiments of the present application, the literature retrieval platform can analyze the user behavior data recorded in the log, obtain the user's retrieval keywords of the user, and use the titles of the literatures corresponding to the literatures whose browsing duration by the user is greater than or equal to the preset duration threshold, the full text is clicked by the user, and the full text is downloaded by the user as the titles of the literatures of interest.

[0064] S22. Use M large language models to process the user's retrieval keywords and the titles of the literatures of interest P times to obtain K recommended keywords.

[0065] Among them, M is a positive integer, and P is a positive integer, but the value of M×P is an integer greater than 1.

[0066] The recommended keyword refers to a recommended word related to the user's retrieval keywords and the titles of the literatures of interest, which is used to reflect the implicit semantics in the user's retrieval keywords and the titles of the literatures of interest. The recommended keywords are used to expand the user's query scope, help the user find literatures in more research fields, and avoid the "information island" in turn. In the embodiments of the present application, the recommended keywords are different from the user's retrieval keywords, so as to provide more retrieval opportunities and discover relevant literatures that were not originally considered.

[0067] The embodiments of the present application use large language models to process the user's retrieval keywords and the titles of the literatures of interest, can utilize the language understanding ability of the large language models, mine the implicit semantic information in the user's retrieval keywords and the titles of the literatures of interest, and recommend the recommended keywords that the user is interested in.

[0068] However, large language models may have hallucination problems, resulting in recommended keywords obtained not meeting the user's query intent. In view of this, embodiments of the present application propose to use M large language models to process the user's retrieved keywords and the titles of the literatures of interest P times to obtain K recommended keywords. For example, using 1 large language model to process the user's retrieved keywords and the titles of the literatures of interest multiple times to obtain K recommended keywords, or using multiple large language models to process the user's retrieved keywords and the titles of the literatures of interest at least once to obtain K recommended keywords. Thus, by using M large language models to process the user's retrieved keywords and the titles of the literatures of interest P times, the hallucination problem caused by a single large language model performing word processing is reduced, and the accuracy of the obtained recommended keywords is improved. It can be understood that the higher the accuracy of the recommended keywords, the more accurately they reflect the user's query intent.

[0069] It can be understood that the larger the value of M, the higher the accuracy of the recommended keywords obtained. However, large language models require computing resources. If the large language model is too large, it will occupy a large amount of computing resources, resulting in slow computing speed, and deploying multiple large language models will also increase the deployment difficulty. Therefore, M should not be too large. In one example, the value of M can be 3.

[0070] Among them, each time each large language model processes the user's retrieved keywords and the titles of the literatures of interest, in order to make the obtained recommended keyword results more accurate, the literature retrieval platform 10 uses the same prompt word each time. The prompt word is an important tool to guide the large language model to generate a specific type of output. By using the same prompt word, the system can more precisely control the output direction of the model, ensuring that the generated recommended keywords meet the user's query intent and requirements. By using the same prompt word, the system can better guide the model to generate reasonable recommended keywords, reducing the occurrence of hallucination problems.

[0071] Further, after obtaining the recommended keywords, the target large language model can be used to process the K recommended keywords based on the judgment prompt word to obtain the target recommended keywords that the user is interested in from the K recommended keywords. Among them, the target large language model belongs to the M large language models.

[0072] Among them, the judgment prompt word is used to judge whether each of the K recommended keywords is a keyword that the user is interested in. For example, the judgment prompt word is "Whether the above recommended keywords are keywords that the user is interested in". By using the multi-round question-and-answer ability of the large language model, the target recommended keywords that the user is more interested in and can better reflect the user's query intent are further obtained.

[0073] In actual use, one of the M large language models can be specified as the target large language model, or a large language model can be randomly assigned as the target large language model based on the M large language models. The embodiments of the present application do not specifically limit this.

[0074] S23. Determine N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database.

[0075] The document information includes information such as the document title, document abstract, and document keywords. For better illustration, the following uses the document title as an example for schematic illustration.

[0076] The target document refers to a document in the database that is highly similar to the recommended keywords. In one example, the target document can be a document whose semantic similarity to the recommended keywords is greater than the preset semantic similarity threshold T. Among them, 0 ≤ T ≤ 1; the larger T is, the more interested the user is in the recommended document. For example, T = 0.8. In another example, the target document is a document whose semantic similarity to each recommended keyword ranks among the top P. P is a positive integer. For example, P = 2.

[0077] In one example, the document retrieval platform 10 calculates the semantic similarity between each of the K recommended keywords and the document information (such as the document title) in the document database, and filters out the documents whose semantic similarity is greater than or equal to T. This document is the target document.

[0078] For example, the recommended keywords generated by the large language model include 3, namely recommended keyword 1: medical image segmentation; recommended keyword 2: application of transfer learning in medical images; recommended keyword 3: application of weakly supervised learning in medical images. The document retrieval platform 10 calculates the semantic similarity between each of the 3 recommended keywords and the document titles in the document database. For example, the following semantic similarity result table is obtained, as shown in Table 1.

[0079] Table 1

[0080]

[0081] If T is 0.8, the target documents include the documents corresponding to Title 1, Title 3, and Title 4.

[0082] In another example, the document retrieval platform 10 obtains the documents whose semantic similarity ranks among the top P for each recommended keyword as the target documents. For example, P = 1. As shown in Table 1, the target document corresponding to recommended keyword 1 is the document corresponding to Title 3, the target document corresponding to recommended keyword 2 is the document corresponding to Title 4, and the target document corresponding to recommended keyword 3 is the document corresponding to Title 4.

[0083] It should be noted that in the embodiments of the present application, for duplicate target documents, they are deduplicated and merged before recommendation. For example, the target documents corresponding to recommended keyword 2 and the target documents corresponding to recommended keyword 3 are both the documents corresponding to title 4. The two target documents are deduplicated, that is, only one document corresponding to title 4 needs to be recommended to the user.

[0084] It should be noted that in the embodiments of the present application, the literature retrieval platform 10 can use the KL algorithm to calculate the semantic similarity between each recommended keyword and the title of the literature in the literature database. The KL-divergence algorithm, also known as the Kullback-Leibler divergence, is an index used to measure the difference between two probability distributions.

[0085] In the embodiments of the present application, if the recommended keyword is a target recommended keyword, the embodiments of the present application calculate the semantic similarity between the target recommended keyword and the literature information in the literature database to determine N target documents, thereby further improving that the target documents can better reflect the user's query intention.

[0086] S24. Recommend N target documents.

[0087] For example, the literature retrieval platform 10 displays the information of the author, title, journal name, year volume issue, etc. of the target document in the reference format on the user data page of the terminal device 101, so as to intelligently and personally recommend the target document to the user.

[0088] In summary, for the literature recommendation method provided in the embodiments of the present application, first, the user's retrieval keywords input on the literature retrieval platform are obtained, as well as the titles of the concerned literatures browsed by the user and retrieved from the literature database based on the user's retrieval keywords, so as to comprehensively reflect the user's intention through the way of context learning. Then, the M large language models are used to process the user's retrieval keywords and the titles of the concerned literatures P times to obtain K recommended keywords, so as to utilize the natural language understanding and generation capabilities of the large language models to mine multiple recommended keywords that are implicit in the user's retrieval keywords and the titles of the concerned literatures and are of interest and value to the user. Among them, the recommended keywords are related to the user's retrieval keywords and the titles of the concerned literatures, but are different from the user's retrieval keywords; each large language model has the same prompt when processing the user's retrieval keywords and the titles of the concerned literatures once, so as to avoid the model output deviation caused by different prompts and further improve the accuracy of the generated recommended keywords. Considering that the large language model may have hallucination problems, resulting in low accuracy of the generated recommended keywords. The embodiments of the present application can use multiple large language models to process the user's retrieval keywords and the titles of the concerned literatures multiple times to eliminate the abnormal recommended keywords caused by the hallucination of the large language model. The user can accurately obtain the target literatures of interest to the user through multiple recommended keywords that can accurately reflect the user's interests, and recommend the target literatures to the user to expand the user's horizons. In summary, through the embodiments of the present application, by using M large language models to process the user's retrieval keywords and the titles of the concerned literatures P times to obtain K recommended keywords, and performing semantic similarity calculation based on the K recommended keywords, the obtained target literatures can broaden the user's literature reading horizons and prevent the user from falling into an "information island". And compared with the recommendation method based on the user's screen, it does not require analyzing a large amount of literature retrieval and reading behavior data of users, so it helps to accurately recommend literatures for users and enhance the universality of literature recommendation. Further, since users are different and the user's retrieval keywords and the titles of the concerned literatures input by users are different, the recommended literatures provided in the embodiments of the present application also have the characteristics of personalization and customization.

[0089] Taking the processing of the user's retrieval keywords and the titles of the concerned literatures by the M large language models once as an example, the literature recommendation method provided in the embodiments of the present application is further described below.

[0090] Appendix Figure 3 is a flowchart of another literature recommendation method provided in the embodiments of the present application. This method is applied to the literature retrieval platform 10 shown in Appendix Figure 1 and includes the following contents:

[0091] ①: Initial data collection.

[0092] In the embodiment of the present application, the literature retrieval platform 10 analyzes user behavior, finds the hit literature based on the user's keywords, and further focuses on the user's browsing and the titles of the literature that the user focuses on as the titles of the literature to be concerned. Among them, the titles of the literature that the user focuses on are the titles of the literature corresponding to the literature for which the user's browsing duration is greater than or equal to the preset duration threshold, and the user clicks on the full text and downloads the full text. It can be obtained by analyzing the user behavior data recorded in the log.

[0093] Among them, the initial data is the user's error-checking keywords and the titles of the literature to be concerned.

[0094] ②: The large language model generates personalized recommended literature keywords.

[0095] Among them, the personalized recommended literature keywords are the recommended keywords.

[0096] The user's retrieval keywords and the titles of the literature to be concerned are a scenario reproduction of the literature and research directions that the user is interested in; using the idea of scenario learning, the user's retrieval keywords and the titles of the literature to be concerned are used as the initial data and sent to the large language model. In the embodiment of the present application, M large language models, namely large language model 1, large language model 2,..., large language model P, are used to process the user's retrieval keywords and the titles of the literature to be concerned respectively. For example, if the quantity is clearly required to be 1 in the prompt, large language model 1 obtains recommended literature keyword 1, large language model 2 obtains recommended literature keyword 2,..., and large language model M obtains recommended literature keyword M.

[0097] Among them, each large language model is processed with the same prompt.

[0098] To avoid the hallucination problem of the large language model, it can be further processed in the following way:

[0099] Step 1: Use M large language models to process the user's retrieval keywords and the titles of the literature to be concerned P times to obtain P×M candidate recommended keywords.

[0100] In the embodiment of the present application, M recommended literature keywords are obtained.

[0101] Step 2: Screen out the keywords to be deleted whose semantic similarity with other candidate recommended keywords is less than the first semantic similarity threshold from the P×M candidate recommended keywords.

[0102] Step 3: Delete the keywords to be deleted from the P×M initial recommended keywords to obtain K1 keywords to be processed.

[0103] Among them, K1 is a positive integer, and K1 is less than or equal to P×M.

[0104] It is understandable that if there are hallucinations in a large language model, there will be significant differences between the recommended literature keywords generated by this large language model and those generated by other language models, and the obtained semantic similarity is relatively small. Therefore, keywords with a semantic similarity less than the first semantic similarity threshold are selected from the M recommended literature keywords, and these keywords are marked as keywords to be deleted. Thus, the recommended literature keywords generated by the hallucinations of the large language model can be removed from the M recommended literature keywords, and K1 keywords to be processed are obtained.

[0105] The first semantic similarity threshold is a preset parameter used to control the semantic similarity between keywords. Usually, this threshold is set relatively low (such as 0.2 or lower) to ensure that the remaining keywords do not contain keywords generated by the hallucinations of the large language model, thereby improving the accuracy of recommendations.

[0106] Step 4: Combine keywords with a semantic similarity greater than or equal to the second semantic similarity threshold among the K1 keywords to be processed, and delete keywords that are the same as the user's retrieved keywords to obtain k recommended keywords.

[0107] Where k is a positive integer and k ≤ K1.

[0108] It is understandable that by de-duplicating and combining, and removing keywords that are the same as the user's retrieved keywords, k recommended keywords are obtained to form a recommended literature keyword list, ensuring that the recommended keywords are both diverse and have a certain difference from the user's initial query, thereby helping the user discover more valuable literature.

[0109] Among them, the second semantic similarity threshold is usually lower than the first semantic similarity threshold and is used to determine which keywords have a high enough semantic similarity to be combined. For example, if the semantic similarity between two keywords is greater than or equal to 0.7, they are considered highly similar and can be combined into one keyword.

[0110] ③: KL-divergePce semantic similarity calculation.

[0111] The literature retrieval platform 10 uses the KL-divergePce algorithm to calculate the vector semantic similarity between each keyword in the recommended literature keyword list and the titles of the literatures in the literature database.

[0112] Furthermore, the literature titles are sorted in descending or ascending order of semantic similarity.

[0113] ④: Personalized intelligent literature recommendation.

[0114] In one example, N documents with semantic similarity greater than a preset semantic similarity threshold T are selected in descending order of semantic similarity and recommended to the user. Here, N is a value preset by those skilled in the art according to needs, and N is a positive integer. For example, N = 20.

[0115] Specifically, the selection method can be as follows: The literature retrieval platform 10 first selects N1 documents with semantic similarity greater than the preset semantic similarity threshold T. If N1 > N, then the documents with the lowest semantic similarity rankings are deleted to obtain N documents recommended to the user. If N1 = N, then the N documents are directly recommended to the user. If N1 < N, then for the top (N - N1) recommended literature keywords that k users are interested in, documents with semantic similarity less than or equal to T are selected, and a total of N documents are recommended to the user.

[0116] In another example, in the embodiment of the present application, there are k recommended literature keywords that users are interested in in the second step. For the semantic similarity between each group of recommended literature keywords and the literature titles, the top |N / K| documents are selected.

[0117] Here, |N / K| represents the result of rounding N / K to the nearest integer.

[0118] If the total number of documents selected here exceeds or is less than N due to rounding, then the corresponding documents are discarded or added according to the semantic similarity, so that the total number of personalized recommended documents is N.

[0119] Specifically, the document retrieval platform 10 obtains information such as the author, title, journal name, and year volume of the documents from the literature database, and recommends the above information to the corresponding users in a targeted and intelligent manner in the reference format on the user data page of the terminal device 101.

[0120] It should be noted that by limiting the number of N recommended to the user in the embodiment of the present application, it is possible to avoid the trouble brought to the user by too much or too little information, help the user obtain more valuable information, and avoid the waste of documents caused by too many documents.

[0121] In summary, the above method uses the idea of scenario learning to let the large language model automatically generate the literature keywords that the user is interested in. By calling multiple large language models, it can not only avoid the risk of incorrect results caused by the hallucination problem of a single large language model, but also accurately generate multiple literature keywords that the user is interested in to expand the user's vision. Finally, the personalized intelligent recommended literature meets the user's expectations and is favored by the user, thereby enhancing the user stickiness of the literature retrieval platform and increasing the access volume of the literature retrieval platform.

[0122] In addition, the embodiment of the present application also provides a literature recommendation device.

[0123] AppendixFigure 4 The structural schematic diagram of the literature recommendation device provided by the embodiment of this application. The device 400 includes:

[0124] An acquisition unit 401, configured to acquire a user's retrieval keyword and the title of the concerned literature; the user's retrieval keyword is the keyword input by the user on the literature retrieval platform, and the title of the concerned literature is the title of the literature corresponding to the literature browsed by the user among the literatures searched from the literature database by the literature retrieval platform based on the user's retrieval keyword;

[0125] A processing unit 402, configured to perform P times of processing on the user's retrieval keyword and the title of the concerned literature by using M large language models to obtain K recommended keywords; M is a positive integer, P is a positive integer, and the product of M and P is an integer greater than 1, K is a positive integer, and K is less than or equal to the product of M and P; the recommended keywords are related to the user's retrieval keyword and the title of the concerned literature, but are different from the user's retrieval keyword; each large language model has the same prompt word for performing one-time processing on the user's retrieval keyword and the title of the concerned literature;

[0126] A recommendation unit 403, configured to determine N target literatures according to the semantic similarity between the K recommended keywords and the literature information in the literature database; recommend N target literatures.

[0127] Among them, the processing unit 402 is specifically configured to:

[0128] Perform P times of processing on the user's retrieval keyword and the title of the concerned literature by using M large language models to obtain P×M candidate recommended keywords; screen out the keywords to be deleted whose semantic similarity threshold with other candidate recommended keywords is less than the first semantic similarity threshold from the P×M candidate recommended keywords; delete the keywords to be deleted from the P×M initial recommended keywords to obtain K1 keywords to be processed; merge the keywords whose semantic similarity is greater than or equal to the second semantic similarity threshold among the K1 keywords to be processed, and delete the keywords that are the same as the user's retrieval keyword to obtain k recommended keywords; the second semantic similarity threshold is greater than the first semantic similarity threshold.

[0129] Optionally, the recommendation unit 403 is specifically configured to: calculate the matching semantic similarity between each of the K recommended keywords and all the literature titles in the literature database; for each recommended keyword, perform the following operations: sort all the literature titles in the literature database according to the matching semantic similarity, and use the literatures corresponding to the literature titles ranked in the top P positions in terms of the matching semantic similarity as the target literatures, where P is a positive integer.

[0130] Optionally, calculate the matching semantic similarity between each of the K recommended keywords and all the document titles in the document database, including: using the KL divergence algorithm to calculate the matching semantic similarity between each of the K recommended keywords and all the document titles in the document database.

[0131] Optionally, the obtaining unit 401 is specifically configured to: analyze the user behavior data recorded in the log; use the document titles corresponding to the documents with the user's browsing time greater than the preset time threshold or the documents clicked and downloaded in full text as the concerned document titles.

[0132] Optionally, the recommending unit 403 is further configured to: display the document authors, titles, journal names, and annual volume and issue numbers of each of the N target documents on the user data page of the document retrieval platform.

[0133] Among them, there are at least two different large language models among the M large language models.

[0134] In summary, the document recommendation device provided by the embodiments of the present application can use multiple large language models to process the user's retrieval keywords and concerned document titles multiple times to eliminate abnormal recommended keywords caused by large language model hallucinations. Through multiple recommended keywords that can accurately reflect the user's interests, the user can accurately obtain the target documents of interest by calculating the semantic similarity, and recommend the target documents to the user to expand the user's horizons. In summary, by using M large language models to process the user's retrieval keywords and concerned document titles P times, the embodiments of the present application obtain K recommended keywords, calculate the semantic similarity based on the K recommended keywords, and the obtained target documents can broaden the user's document reading horizons and prevent the user from falling into an "information island". Moreover, compared with the recommendation method based on the user's profile, it is not necessary to analyze a large amount of user's document retrieval and reading behavior data, so it helps to accurately recommend documents for users and enhance the universality of document recommendation.

[0135] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions that can run on a computing device or be stored in any available medium. When the computer program product runs on a computing device, it causes the computing device to execute the above-mentioned document recommendation method. The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive), etc. The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned document recommendation method.

[0136] The descriptions of the processes or structures corresponding to the above respective drawings have different focuses. For the parts not detailed in a certain process or structure, reference can be made to the relevant descriptions of other processes or structures.

[0137] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claimed rights.

Claims

1. A literature recommendation method, characterized in that, The method includes: Obtain the user's retrieval keywords and the titles of the concerned documents; the user's retrieval keywords are the keywords input by the user on the document retrieval platform, and the titles of the concerned documents are the titles of the documents corresponding to the documents that the user has browsed after clicking on the details among the documents searched by the document retrieval platform from the document database based on the user's retrieval keywords; Use M large language models to perform P times of processing on the user's retrieval keywords and the titles of the concerned documents to obtain P×M candidate recommended keywords; screen out the keywords to be deleted whose semantic similarity threshold with other candidate recommended keywords is less than the first semantic similarity threshold from the P×M candidate recommended keywords; delete the keywords to be deleted from the P×M candidate recommended keywords to obtain K1 keywords to be processed; merge the keywords with a semantic similarity greater than or equal to the second semantic similarity threshold among the K1 keywords to be processed, and delete the keywords that are the same as the user's retrieval keywords to obtain K recommended keywords; the second semantic similarity threshold is greater than the first semantic similarity threshold; M is equal to 3, P is a positive integer, K is a positive integer, and K is less than or equal to the product of M and P; each large language model performs each processing on the user's retrieval keywords and the titles of the concerned documents based on the same prompt; Determine N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database; recommend the N target documents; N is a positive integer.

2. The method according to claim 1, wherein The method further includes: Use the target large language model to process the K recommended keywords based on the judgment prompt to obtain the target recommended keywords that the user is interested in from the K recommended keywords; Wherein, the judgment prompt is used to judge whether each recommended keyword among the K recommended keywords is a keyword that the user is interested in; the target large language model belongs to the M large language models; The determining N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database includes: Determine N target documents according to the semantic similarity between the target recommended keywords and the document information in the document database.

3. The method according to claim 1, characterized in that, The determining N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database includes: Calculate the matching semantic similarity between each recommended keyword among the K recommended keywords and all the document titles in the document database; For each recommended keyword, perform the following: sort all the document titles in the document database according to the matching semantic similarity, and take the documents corresponding to the document titles ranked in the top P positions in terms of the matching semantic similarity as the target documents, where P is a positive integer.

4. The method according to claim 3, wherein The calculating the matching semantic similarity between each recommended keyword among the K recommended keywords and all the document titles in the document database includes: Using the KL divergence algorithm, calculate the matching semantic similarity between each of the K recommended keywords and all the document titles in the document database.

5. The method according to claim 3, wherein Obtain the concerned document titles, including: Analyze the user behavior data recorded in the log; Take the document titles corresponding to the documents whose browsing time by the user is greater than the preset time threshold, or the documents for which the full text has been clicked and downloaded, as the concerned document titles.

6. The method according to claim 1, wherein The recommending of the N target documents includes: Display the document author, title, journal name, and / or year / volume / issue of each of the N target documents on the user data page of the document retrieval platform.

7. A literature recommendation device, characterized in that, The device includes: An acquisition unit, configured to acquire a user retrieval keyword and concerned document titles; the user retrieval keyword is the keyword input by the user on the document retrieval platform, and the concerned document titles are the document titles corresponding to the documents browsed by the user among the documents searched from the document database by the document retrieval platform based on the user retrieval keyword; A processing unit, configured to use M large language models to perform P times of processing on the user retrieval keyword and the concerned document titles to obtain P×M candidate recommended keywords; screen out the keywords to be deleted whose semantic similarity threshold with other candidate recommended keywords is less than the first semantic similarity threshold from the P×M candidate recommended keywords; delete the keywords to be deleted from the P×M candidate recommended keywords to obtain K1 keywords to be processed; merge the keywords with a semantic similarity greater than or equal to the second semantic similarity threshold among the K1 keywords to be processed, and delete the keywords identical to the user retrieval keyword to obtain K recommended keywords; the second semantic similarity threshold is greater than the first semantic similarity threshold; M is equal to 3, P is a positive integer, K is a positive integer, and K is less than or equal to the product of M and P; each large language model performs each processing on the user retrieval keyword and the concerned document titles based on the same prompt; A recommendation unit, configured to determine N target documents according to the semantic similarity between the K recommended keywords and the document information in the document database; recommend the N target documents.

8. A computer storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause the computer to execute the document recommendation method according to any one of claims 1 to 6.

Images