Literature evaluation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a literature evaluation method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a target domain, an evaluation dimension and a to-be-evaluated document input by a user; searching first information which belongs to the target domain and corresponds to the evaluation dimension in a target time period from the Internet; retrieving a literature library to obtain second information associated with the literature to be evaluated; obtaining third information associated with the to-be-evaluated literature from a pre-constructed knowledge graph corresponding to the target domain; according to the first information, the second information, the third information, the target domain and the evaluation dimension, constructing to obtain a model cue word; and inputting the to-be-evaluated literature and the model cue word into a large language model for literature evaluation, so that the large language model outputs an evaluation result of the to-be-evaluated literature in the evaluation dimension and evaluation basis information corresponding to the evaluation result.

Description

Technical Field

[0001] This application relates to the field of intelligent document evaluation technology, and in particular to a document evaluation method, device, electronic device and storage medium. Background Technology

[0002] In recent years, with the development and popularization of artificial intelligence (AI) technology, automated scientific metrology and evaluation has become one of the important research directions in the field of AI. AI technologies based on machine learning and large models can help people complete specific tasks more quickly and accurately, significantly reducing labor costs and improving work efficiency.

[0003] How to achieve the accuracy and interpretability of automatic evaluation of documents such as patents or papers is an urgent problem to be solved. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of this application is to provide a literature evaluation method, apparatus, electronic device and storage medium to effectively improve the accuracy and interpretability of literature evaluation.

[0005] In a first aspect, embodiments of this application provide a literature evaluation method, the method comprising: Obtain the target domain, evaluation dimensions, and literature to be evaluated as input by the user; Retrieve first information from the internet that belongs to the target domain and corresponds to the evaluation dimension within the target time period; Second information related to the document to be evaluated is retrieved from the literature database; Obtain third information associated with the document to be evaluated from the pre-constructed knowledge graph corresponding to the target domain; Based on the first information, the second information, the third information, the target domain, and the evaluation dimension, model prompt words are constructed; The document to be evaluated and the model prompts are input into a large language model for document evaluation, so that the large language model outputs the evaluation results of the document to be evaluated in the evaluation dimension and the evaluation basis information corresponding to the evaluation results.

[0006] Secondly, embodiments of this application provide a document evaluation apparatus, the apparatus comprising: The acquisition module is used to acquire the target domain, evaluation dimensions, and literature to be evaluated, as input by the user. The first information retrieval module is used to retrieve first information from the Internet that belongs to the target domain and corresponds to the evaluation dimension within the target time period. The second information retrieval module is used to retrieve second information associated with the document to be evaluated from the document database; The third information acquisition module is used to acquire third information associated with the document to be evaluated from the pre-constructed knowledge graph corresponding to the target domain. The prompt word construction module is used to construct model prompt words based on the first information, the second information, the third information, the target domain, and the evaluation dimension; The document evaluation module is used to input the document to be evaluated and the model prompt words into a large language model for document evaluation, so that the large language model outputs the evaluation result of the document to be evaluated in the evaluation dimension and the evaluation basis information corresponding to the evaluation result.

[0007] Thirdly, embodiments of this application provide an electronic device, including: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements any of the preceding literature evaluation methods.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium that, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the document evaluation methods described above.

[0009] Compared with the prior art, the embodiments of this application have the following advantages: In this embodiment, by integrating the first information from internet retrieval, the second information from the literature database, and the third information from the target domain knowledge graph, and combining the target domain and evaluation dimensions, model prompt words are accurately constructed. After inputting the literature to be evaluated and the prompt words into the large language model, the multi-source information provides a comprehensive and professional basis for the evaluation. At the same time, the prompt words clarify the core dimensions and domain logic of the evaluation, making the evaluation results output by the large language model more realistic and traceable, thereby effectively improving the accuracy and interpretability of literature evaluation.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0011] Figure 1 A flowchart illustrating the steps of a document evaluation method provided in this application embodiment; Figure 2 A flowchart illustrating the steps of a first information acquisition method provided in this application embodiment; Figure 3 A flowchart illustrating the steps of a second information acquisition method provided in this application embodiment; Figure 4 A flowchart illustrating the steps of a knowledge graph construction method provided in this application embodiment; Figure 5 A flowchart illustrating the steps of a third information acquisition method provided in this application embodiment; Figure 6 A flowchart illustrating the steps of a target document screening method provided in this application embodiment; Figure 7 This is a schematic diagram of the structure of a document evaluation device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0013] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0014] Reference Figure 1 The flowchart illustrates the steps of a literature evaluation method provided in an embodiment of this application, such as... Figure 1 As shown, the literature evaluation method may include steps 101 to 106.

[0015] Step 101: Obtain the target domain, evaluation dimensions, and literature to be evaluated as input by the user.

[0016] The embodiments of this application can be applied to scenarios where documents are intelligently evaluated based on large models and knowledge graphs.

[0017] The target field refers to the professional category to which the literature to be evaluated belongs, as specified by the user. It can be a specific technical field (such as autonomous driving, biomedicine) or a subject field (such as computer science, materials engineering).

[0018] Evaluation dimensions refer to the core directions of literature evaluation set by users, which can cover academic value dimensions (such as innovation points and citation influence), economic value dimensions (such as industrialization potential and application scenarios), and social value dimensions (such as industry promotion role), and are the core orientation of evaluation results.

[0019] The documents to be evaluated refer to documents for which the user needs to assess their value. In this embodiment, the documents to be evaluated may include: academic papers, patents, and other documents.

[0020] When a user needs to evaluate a document, the user can input the target field, evaluation dimensions, and the document to be evaluated.

[0021] In its implementation, the system can provide user interaction entry points (such as text input boxes and document upload ports), allowing users to manually input the target field name (e.g., the medical field) and evaluation dimensions (e.g., academic innovation value). Users can also upload documents to be evaluated or input document titles, and the system automatically retrieves the full text. The system verifies the clarity of the target field description (prompting users to refine it if ambiguous), checks if the evaluation dimensions conform to preset ranges (or allows users to define and supplement dimension definitions), and verifies if the document to be evaluated can be parsed correctly (prompting users to re-upload if the format is incorrect). Finally, the system can store verified user input information, generate an evaluation task number, and link it to all subsequent operation data to ensure process traceability.

[0022] Step 102: Retrieve first information from the Internet that belongs to the target domain and corresponds to the evaluation dimension within the target time period.

[0023] The first information refers to publicly available technical information retrieved from the Internet that belongs to the target field within the target time period and corresponds to the evaluation dimension. It is technical data with high timeliness in the field, including but not limited to the latest developments in the field's technology, industry technology development reports, publicly available technical achievements of enterprises, pre-published content of academic conferences, open discussions in technical communities, and other structured / semi-structured information, papers, patents, etc.

[0024] The target time period refers to the information retrieval time range that is user-defined or system default. It is used to filter the latest literature, the latest technological development trends, technology descriptions, and other information in the target field to ensure the timeliness of the evaluation and reference.

[0025] After obtaining the target domain and evaluation dimensions input by the user, the first information belonging to the target domain and corresponding to the evaluation dimensions within the target time period can be retrieved from the internet. Specifically, a multi-dimensional combined search keyword set can be constructed based on a standardized core keyword set for the target domain and evaluation dimension names / codes, and the keywords can be expanded (synonyms, near-synonyms, and domain-related technical terms). Then, the constructed keyword set is input into the selected search channel, the search operation is executed, and the search results are precisely filtered based on the target time period to remove invalid information outside the time period, so as to obtain the first information. The retrieval process for the first information will be combined with the following embodiments. Figure 2 A detailed description will not be repeated here.

[0026] Step 103: Retrieve second information associated with the document to be evaluated from the literature database.

[0027] The second type of information refers to publicly available technical literature information retrieved from professional literature databases that has technical characteristics related to the document to be evaluated. This information includes similar / related comparative literature information, such as the full text / core abstract, technical characteristics, basic attributes of the comparative literature (publication date, author, journal / patent office, etc.), and existing evaluation conclusions.

[0028] After receiving the document to be evaluated input by the user, second information associated with the document can be retrieved from the document database. Specifically, technical features of the document to be evaluated can be extracted first to obtain a core set of technical features, which serves as the core matching basis for the document database retrieval. Information retrieval is then performed on the document database based on the core set of technical features to obtain the second information. The specific retrieval process for the second information will be discussed in the following embodiments. Figure 3 A detailed description will not be repeated here.

[0029] Step 104: Obtain third information associated with the document to be evaluated from the pre-constructed knowledge graph corresponding to the target domain.

[0030] A knowledge graph is a pre-constructed domain-specific technical knowledge graph. It uses technical entities and literature evaluation metrics as core nodes, technical relationships between nodes as edges, and literature attribute information as node attributes. It also enables the tracing and association of graph nodes / edges with the original literature, serving as a structured representation of technical knowledge in the target domain. The knowledge graph construction process will be discussed in the following embodiments. Figure 4 A detailed description will not be repeated here.

[0031] The third information refers to the structured technical knowledge information related to the document to be evaluated, obtained from the knowledge graph of the target domain. This includes nodes in the knowledge graph that match the document to be evaluated, node attributes, relationships between nodes, as well as newly added nodes / relationships and newly identified nodes / relationships after the document to be evaluated updates the knowledge graph.

[0032] After receiving the document to be evaluated from the user input, third-party information associated with the document can be obtained from a pre-constructed knowledge graph of the target domain. Specifically, information can be extracted from the document to be evaluated first, and the knowledge graph can be updated with the extracted information. Then, the third-party information can be extracted based on the updated knowledge graph. This implementation process will be combined with the following embodiments. Figure 5 A detailed description will not be repeated here.

[0033] Step 105: Construct model prompt words based on the first information, the second information, the third information, the target domain, and the evaluation dimension.

[0034] Model prompts refer to standardized input prompts built for a large language model for literature evaluation. They integrate the target domain, evaluation dimensions, first information, second information, third information, and evaluation requirements of the literature to be evaluated. They serve as instructions and information carriers for the large language model to evaluate literature and need to be adapted to the understanding logic of the large language model.

[0035] After obtaining the first, second, and third pieces of information, model prompts can be constructed based on these, the target domain, and the evaluation dimensions. Specifically, a prompt framework specific to technical literature evaluation can be used, adapted to the reasoning logic of the large-scale language model for literature evaluation. The first, second, and third pieces of information, the target domain, and the evaluation dimensions are then filled into the prompt framework as needed. Natural language optimization is performed on the filled prompts to ensure fluency and logical coherence. Professional expressions used in technical literature evaluation are also employed to match the training corpus style of the large-scale language model for literature evaluation, improving the model's information comprehension efficiency. The initially constructed prompts are input into a small-sample test model for validity verification. If the model cannot accurately recognize the instructions / information, the prompts are adjusted in terms of expression and information organization until the verification passes, yielding the final model prompts.

[0036] Step 106: Input the document to be evaluated and the model prompt words into the large language model used for document evaluation, so that the large language model outputs the evaluation result of the document to be evaluated in the evaluation dimension and the evaluation basis information corresponding to the evaluation result.

[0037] The large language model is a pre-trained large language model optimized for technical document evaluation scenarios. It has the ability to understand technical documents, extract technical features, perform multi-dimensional evaluation analysis, and reason about evaluation criteria. It can receive input from the document to be evaluated and model prompts, and output structured evaluation results and evaluation criteria.

[0038] The evaluation results refer to the quantitative / qualitative evaluation results output by the large language model for the literature to be evaluated under the evaluation dimensions specified by the user. The quantitative results adopt a standardized scoring system (such as 1-10 points, star rating), and the qualitative results adopt a level description (such as high / medium / low, excellent / good / average / poor).

[0039] The evaluation basis information refers to the reasoning and supporting basis given simultaneously when the large language model outputs the evaluation results. It is strongly related to the first, second and third information, including but not limited to the characteristics of the related literature for comparison, the matching of the frontier of the field technology, the technical relationship logic in the knowledge graph, and the matching process of the evaluation criteria of the evaluation dimensions.

[0040] After obtaining the model prompts, the prompts and the document to be evaluated can be input into a large language model for document evaluation. This allows the large language model to output the evaluation results for the document across evaluation dimensions and the corresponding evaluation basis information. Specifically, the structured text of the document to be evaluated and the model prompts can be converted into an input data format supported by the large language model (e.g., JSON format, containing "prompt" and "document" fields, corresponding to the model prompts and the document to be evaluated, respectively). The converted input data is then input into the large language model through the model inference interface, triggering the model's evaluation inference calculation. Based on the input prompts, the large language model first fully understands the core technical content of the document to be evaluated. Then, combining the first, second, and third information in the prompts, it performs multi-dimensional evaluation inference. The specific process is as follows: for each evaluation dimension, it matches the reference information with the technical features of the document to be evaluated one by one. Based on the matching results, it performs quantitative / qualitative evaluation according to preset evaluation standards, analyzes the supporting logic of the reference information (i.e., the aforementioned first, second, and third information) for the evaluation results, clarifies the corroborating role of each piece of information in the evaluation results, and forms detailed evaluation basis.

[0041] This application embodiment integrates first information from internet retrieval, second information from literature database association, and third information from the target domain knowledge graph. By combining the target domain and evaluation dimensions, it accurately constructs model prompt words. After inputting the literature to be evaluated and the prompt words into the large language model, the multi-source information provides a comprehensive and professional basis for the evaluation. At the same time, the prompt words clarify the core dimensions and domain logic of the evaluation, making the evaluation results output by the large language model more realistic and traceable, thereby effectively improving the accuracy and interpretability of literature evaluation.

[0042] Next, combined Figure 2 The process of retrieving the first information is described in detail.

[0043] Reference Figure 2 The diagram illustrates a flowchart of a first information acquisition method provided in an embodiment of this application. Figure 2 As shown, the first information acquisition method may include steps 201 to 204.

[0044] Step 201: Obtain the target time of the most recent update of the knowledge graph.

[0045] In this embodiment, the target time refers to the specific time point at which the pre-constructed target domain knowledge graph last completed data updates and synchronized archiving of node / edge information, accurate to the hour, minute, and second. It serves as the starting benchmark for defining the preset duration and is extracted from the knowledge graph's update log.

[0046] When retrieving the first piece of information, the target time of the most recent update of the knowledge graph can be obtained. Specifically, the "completion time" of the most recent graph update can be extracted by filtering conditions (sorting in descending order of update completion time and taking the first record) and determined as the target time.

[0047] Step 202: Determine the time period between the target time and the current time as the target time period.

[0048] The current moment refers to the moment at which the literature to be evaluated is being evaluated.

[0049] After obtaining the target time, the time interval between the target time and the current time can be obtained to determine the target time interval.

[0050] Step 203: Retrieve from the Internet literature and technical description information that belong to the target field and correspond to the evaluation dimension within the target time period.

[0051] After determining the target time period, literature and technical descriptions belonging to the target field and corresponding evaluation dimensions can be retrieved from the internet within that time period. In other words, using the target time period, target field, and evaluation dimensions as search criteria, information such as literature and technological development trends can be retrieved from the internet to obtain the final literature and technical descriptions.

[0052] This application's embodiment uses the most recent update time of the knowledge graph as the starting benchmark for the target time period, ensuring that the retrieval time range of the first information is precisely synchronized with the knowledge graph update cycle. This ensures that the retrieved information is all the latest achievements in the field added after the knowledge graph update, supplementing the new knowledge not covered by the graph, and avoids the problem of outdated or omitted literature due to subjective setting of the target time period. This provides a more timely and targeted reference for the construction of model prompt words, further improving the accuracy and rationality of the literature evaluation results.

[0053] Next, combined Figure 3 The process of retrieving the second piece of information is described in detail.

[0054] Reference Figure 3 The diagram illustrates a flowchart of a second information acquisition method provided in an embodiment of this application. Figure 3 As shown, the second information acquisition method may include steps 301 to 303.

[0055] Step 301: Extract the first literature information from the literature to be evaluated; wherein the first literature information includes at least: the literature abstract, research purpose, methods used to solve the problem, evaluation indicators and evaluation set used in the research, and research conclusions of the literature to be evaluated.

[0056] In this embodiment, the first document information refers to the core structured research information extracted from the document to be evaluated. It is the benchmark information for document similarity matching and includes at least the document abstract, research purpose, methods used to solve the problem, evaluation indicators and evaluation sets used in the research, and research conclusions of the document to be evaluated. It is the core research feature carrier of the document to be evaluated.

[0057] After obtaining the literature to be evaluated, the primary information can be extracted from it. Specifically, based on natural language processing technology, combined with domain-optimized text classification and entity extraction models, targeted and precise extraction can be performed according to preset dimensions (literature abstract, research objective, methods used to solve the problem, evaluation indicators and assessment sets used in the research, and research conclusions), as follows: Abstract: Extract the formal abstract that comes with the document directly. If there is no abstract, use the abstract generation model to generate a core abstract based on the full text (within 300 words).

[0058] Research objective: To extract and streamline textual content from the literature that clearly states the research objectives, significance, and core technical issues to be addressed.

[0059] The method used to solve the problem was to extract the textual content of the core technical methods, technical solutions, experimental procedures, algorithm models, and other specific implementation means from the literature.

[0060] The evaluation metrics and evaluation sets used in the study: quantitative evaluation metrics (such as precision, recall, and F1 score) and evaluation datasets / experiment sets (such as public datasets and self-built evaluation sets) were extracted from the literature to verify the study effect.

[0061] Research conclusions: The core content of the research results analysis, experimental conclusions, technical effect summary, and research innovations and shortcomings were extracted from the literature.

[0062] Finally, the extracted information from each dimension can be structured and stored in the form of "dimensional label + core content" (such as JSON format) to ensure that the information from each dimension is independent, clear, without redundancy or omissions, thus forming standardized first-reference information.

[0063] Step 302: Obtain the second literature information of each document in the literature database.

[0064] The second literature information refers to the core structured research information extracted from each publicly available document in the literature database that is completely corresponding to the first literature information dimension. It is the basic information for comparing similarity with the document to be evaluated. The extraction dimension is consistent with the first literature information to ensure the comparability of similarity calculation.

[0065] For a pre-established literature database, all documents in the database can be batch formatted, segmented into text, and uniformly converted into plain text structured documents. The processing rules are completely consistent with the preprocessing rules of the documents to be evaluated in step 301, ensuring that the data format is consistent.

[0066] Step 303: Based on the comprehensive similarity between the first document information and the second document information, select the second information from the second document information whose comprehensive similarity is greater than the similarity threshold.

[0067] Comprehensive similarity refers to the overall matching degree obtained by weighting the multi-dimensional sub-similarity based on the information of the first document and the information of a single second document, rather than the similarity of a single dimension. It is the core quantitative indicator for measuring the degree of correlation between the document to be evaluated and the research characteristics of the documents in the document database, and the value range is [0,1] or [0,100%].

[0068] The similarity threshold refers to a pre-set comprehensive similarity screening threshold, which is set by a person skilled in the art based on the technical characteristics of the target field and the needs of document evaluation (e.g., 80%). It is a quantitative standard for judging whether a document in the document database is related to the document to be evaluated.

[0069] After obtaining the first and second document information, a comprehensive similarity can be calculated between them, and the second document information can be selected from the second document information based on this comprehensive similarity. Specifically, based on the technical characteristics of the target field and the document evaluation requirements, differentiated weights (with a weight sum of 1) can be assigned to the five core dimensions of the first and second document information (i.e., the abstract of the document to be evaluated, the research purpose, the methods used to solve the problem, the evaluation indicators and evaluation set used in the research, and the research conclusions), with higher weights for the core dimensions. Then, a text similarity calculation algorithm is used to calculate the sub-similarity of the first document information and a single second document information in the five core dimensions, obtaining a five-dimensional sub-similarity set for a single document (e.g., abstract similarity 0.85, research method similarity 0.90), with sub-similarity values ​​ranging from [0,1]. Finally, a comprehensive similarity can be calculated: based on the preset weights of each dimension, the five-dimensional sub-similarity of a single document is weighted and summed to obtain the comprehensive similarity between the first document information and the second document information.

[0070] After obtaining the overall similarity score, all secondary document information can be sorted from high to low based on the overall similarity score. The sorted secondary document information is then compared with a preset similarity threshold. Secondary document information with an overall similarity score greater than the similarity threshold is selected and used as secondary information related to the documents to be evaluated. The unique identifier of each document is retained, enabling tracing back to the original documents in the document database.

[0071] This application's embodiments overcome the limitations of single-dimensional similarity by employing a multi-dimensional sub-similarity weighted calculation to achieve comprehensive similarity. This approach aligns with the research characteristics of technical literature and more accurately reflects the actual degree of correlation between documents. Through quantitative screening using similarity thresholds, automated and efficient screening of related documents in the literature database is achieved, avoiding the subjectivity and inefficiency of manual screening and ensuring the consistency and effectiveness of the screening results. Simultaneously, the secondary information obtained from the screening provides a highly relevant and reliable comparative basis for the subsequent evaluation of the documents to be evaluated, laying a data foundation for improving the accuracy of the literature evaluation results.

[0072] Next, combined Figure 4 The implementation process of constructing a knowledge graph is described in detail.

[0073] Reference Figure 4 The diagram illustrates a flowchart of the steps involved in constructing a knowledge graph according to an embodiment of this application. Figure 4 As shown, the knowledge graph construction method may include steps 401 to 403.

[0074] Step 401: Obtain published literature belonging to the specified field; wherein the specified field includes at least the target field.

[0075] The designated domain refers to the boundary of the knowledge graph construction scope. It is a professional category that covers the target domain (the domain that users focus on for evaluation) and extended related domains. It can be dynamically adjusted according to the scope of literature coverage to ensure the domain integrity of the graph.

[0076] Published documents refer to various publicly available documents belonging to a specific field, including academic papers, patent documents (published texts of invention / utility model patents), industry research reports, etc., which are the core source of map data.

[0077] When constructing a knowledge graph, published literature belonging to a specific field can be obtained. Specifically, a search can be conducted from multi-source literature databases (such as academic paper databases, patent databases, authoritative industry literature platforms, etc.) using a specified field as the search criterion to obtain published literature.

[0078] Step 402: Obtain the literature evaluation indicators, technical entity terms, literature relationship information, and literature attribute information associated with the literature evaluation indicators and technical entity terms of the published literature; wherein, the literature relationship information includes at least: information on the technical means used and the technical problems solved by the published literature, and the literature attribute information is structured information associated with the literature evaluation indicators and technical entity terms.

[0079] Literature evaluation indicators refer to core indicators used to quantitatively or qualitatively evaluate the technical value and research quality of published literature, including but not limited to accuracy.

[0080] Technical entity terms refer to words extracted from published literature that represent the core technical elements of a specific field. These include method-related terms (such as "BERT entity recognition algorithm"), device-related terms (such as "graph database"), and parameter-related terms (such as "similarity threshold 80%)", which are the core components of knowledge graph nodes.

[0081] Document relationship information refers to structured data that is directly related to document evaluation indicators and technical entities. It is a supplementary explanation of node information, including but not limited to technical sub-directions, the basis for calculating evaluation indicators, and the application scenarios of technical entities in the document. It cannot be unstructured or irrelevant information.

[0082] After obtaining published literature, we can acquire its evaluation metrics, technical entity terms, literature relationship information, and literature attribute information associated with these metrics and technical entity terms. Specifically, we can use domain-optimized natural language processing tools, including entity recognition models (BERT-NER), text classification models, and relation extraction models, combined with a specified domain technical dictionary, to ensure the accuracy of information extraction. Furthermore, for the preprocessed published literature, we can extract information in batches according to the following rules to ensure the completeness of the information dimensions for each document: Literature evaluation indicators: Extract evaluation indicators explicitly stated in the literature, or calculate quantitative indicators (such as citation count and novelty score) based on the research content and technical effects of the literature according to the evaluation standards of the specified field, and organize them into a standardized indicator format.

[0083] Technical entity words: Using entity recognition algorithms such as BERT-NER, core technical entity words such as method, equipment, and parameter words in the documents are extracted. After being verified and denoised by a specified domain technical dictionary, non-technical words are removed to form a set of technical entity words for each document.

[0084] Document Relationship Information: The focus is on extracting the "technical means" and "technical problems solved" recorded in the documents. A relation extraction model is used to identify the corresponding relationship between the two. At the same time, other technical relationship information can be extracted to ensure that at least the core relationship information is covered. In this embodiment, the document relationship information includes at least: information on the technical means used and the technical problems solved in the published documents, and the document attribute information is structured information associated with document evaluation indicators and technical entity terms.

[0085] Document attribute information: Only structured information related to document evaluation indicators and technical entities is extracted, such as the calculation basis of evaluation indicators, application scenarios of technical entities, and publication date of documents. Irrelevant unstructured information is filtered out and organized into a structured manner by dimension.

[0086] Step 403: Using the literature evaluation indicators and the technical entity words as graph nodes of the knowledge graph, the literature attribute information is associated with each graph node as the node attribute information of the corresponding graph node. Each graph node and each graph edge is uniquely labeled and associated with the corresponding published literature to construct the knowledge graph of the specified field.

[0087] Graph nodes are the basic building blocks of a knowledge graph. Specifically, they refer to nodes formed around document evaluation indicators and technical entity terms, which are used to carry core technical information in a specified field.

[0088] Graph edges are the association carriers that connect various graph nodes in a knowledge graph. They are used to represent the technical associations between different graph nodes (such as "application associations of literature evaluation indicators and technical entity words" and "dependency associations between technical entity words").

[0089] Unique identifier association refers to binding a unique identifier (such as a patent number or document DOI) to each graph node and edge of the knowledge graph, which corresponds to a published document. This enables a one-to-one correspondence between graph nodes and edges and the original published documents, facilitating subsequent association retrieval and verification.

[0090] After acquiring the above information, all extracted literature evaluation indicators and technical entity terms can be used as core nodes of the knowledge graph. Duplicate nodes (such as the same technical entity term appearing in different documents) are deduplicated and merged. A unique node identifier is assigned to each node, clarifying its type ("evaluation indicator" or "technical entity term"). The extracted literature attribute information is then associated with each graph node according to the "attribute-node" correspondence, serving as the node's attribute information (e.g., associating "evaluation indicator calculation basis" with the corresponding evaluation indicator node, and "technical entity term application scenario" with the corresponding technical entity term node), ensuring accurate matching between attribute information and nodes. Simultaneously, based on the extracted literature relationship information, graph edges connecting each graph node can be constructed, clarifying the semantic association type of each edge (e.g., "technical entity term - technical means association" or "evaluation indicator - technical entity term application association"), and assigning a unique edge identifier to each edge.

[0091] Each graph node and each graph edge is bound to a unique identifier of the corresponding published literature (such as patent number or DOI number), enabling the nodes and edges to correspond to the original literature and facilitating the rapid location of the literature source during subsequent searches.

[0092] This application embodiment accurately collects published literature in a specified field, extracts core structured technical information, and constructs graph nodes, associated attribute information, and graph edges based on literature evaluation indicators and technical entity terms. It also realizes the source association between the graph and the original literature. The constructed knowledge graph can comprehensively cover the core technical information of the specified field (including the target field), and structurally integrates the technical elements and relationships in the literature, avoiding the fragmentation of technical information. This not only ensures the integrity and accuracy of the knowledge graph, but also provides reliable structured knowledge support for the rapid and accurate acquisition of related information of the literature to be evaluated, thereby improving the efficiency and accuracy of the literature evaluation process.

[0093] Next, combined Figure 5 The process of obtaining third-party information is described in detail.

[0094] Reference Figure 5 The diagram illustrates a flowchart of a third information acquisition method provided in an embodiment of this application. Figure 5 As shown, the third information acquisition method may include steps 501 to 503.

[0095] Step 501: Obtain the target document evaluation index, target technical entity words, target document relationship information, and target document attribute information associated with the target document evaluation index and the target technical entity words.

[0096] In this embodiment, the target literature evaluation index is a core indicator extracted from the literature to be evaluated and used to quantitatively or qualitatively evaluate the technical value and research quality of the literature. It is consistent with the extraction dimensions of the "literature evaluation index" of published literature (such as technical novelty score, practicality score, etc.) and is one of the core bases for updating the knowledge graph and matching related nodes.

[0097] Target technical entity words refer to words extracted from the literature to be evaluated that represent the core technical elements of the literature. They are consistent with the classification of "technical entity words" in published literature (method, equipment, parameter, etc.), and are verified and denoised by the domain technical dictionary. They are used for node updates and association matching in the knowledge graph.

[0098] Target document relationship information refers to the association information used to characterize the technical logic of the document to be evaluated. The core includes the technical means used by the document to be evaluated and the technical problems solved. It is consistent with the "document relationship information" dimension of published documents and is used for edge updates of the knowledge graph.

[0099] Target document attribute information refers to structured information that is directly related to the target document evaluation indicators and target technical entity words of the document to be evaluated. It is consistent with the extraction rules of the "document attribute information" of published documents (such as the calculation basis of evaluation indicators, application scenarios of technical entity words, etc.) and is used to supplement the attribute information of knowledge graph nodes.

[0100] After obtaining the literature to be evaluated, it can be cleaned (watermarks, blank lines, and irrelevant formatting removed) and converted into plain text structured text. The text is then divided into modules according to its natural paragraphs (abstract, research methods, conclusions, etc.) to facilitate targeted information extraction. Extraction tools and rules are calibrated: The same information extraction tools (domain-optimized BERT-NER entity recognition model, relation extraction model, etc.) and extraction rules used in constructing the original target domain knowledge graph are employed to ensure that the various target information in the literature to be evaluated is consistent in dimension and format with the original graph, avoiding incompatibility issues during updates. Targeted extraction of various target information is then performed. Evaluation indicators for target literature: Extract evaluation indicators that are explicitly stated in the literature to be evaluated. If the literature does not explicitly state them, calculate quantitative indicators based on the evaluation standards of the target field, combined with the research content and technical effects of the literature, and organize them into a standardized format.

[0101] Target technical entity words: Core technical terms such as methods, equipment, and parameters in the literature are extracted using an entity recognition model. After verification and noise reduction by the target domain technical dictionary, non-technical terms are removed to form a set of target technical entity words.

[0102] Target document relationship information: Focus on extracting the technical means and technical problems solved by the documents to be evaluated, and use a relationship extraction model to identify the corresponding relationship between the two to ensure coverage of core relationship information. Other relevant technical relationship information can be extracted as a supplement.

[0103] Target document attribute information: Only extract structured information related to the above two types of target information (evaluation indicators and technical entity terms), such as the calculation basis of evaluation indicators, the application scenarios of technical entity terms in the document, and the core research direction of the document, filter out irrelevant unstructured information, and organize it into a structured manner according to dimensions.

[0104] Finally, the extracted four types of target information can be verified to correct extraction errors and supplement missing information, ensuring that the information is complete and free of redundancy. The four types of information are then linked and integrated in the form of "document identifier to be evaluated + information dimension" to form a standardized set of target information for the documents to be evaluated.

[0105] Step 502: Update the knowledge graph corresponding to the target domain based on the target document evaluation indicators, the target technical entity words, the target document relationship information, and the target document attribute information to obtain the target knowledge graph.

[0106] A target knowledge graph is a new version of a knowledge graph that is updated by incorporating various target information (target document evaluation indicators, target technical entity words, etc.) of the document to be evaluated from a pre-constructed target domain knowledge graph. It retains the core technical information of the original graph and adds the relevant technical elements and relationships of the document to be evaluated.

[0107] After obtaining the target document evaluation metrics, target technical entity terms, target document relationship information, and target document attribute information, the knowledge graph corresponding to the target domain can be updated based on these metrics to obtain an updated target knowledge graph. Specifically, a pre-constructed target domain knowledge graph can be retrieved from a graph database to obtain all nodes (document evaluation metrics, technical entity terms), node attributes, graph edges (relationships), and the labeled association information between nodes / edges and published documents. The target document evaluation metrics and target technical entity terms of the document to be evaluated are then matched and compared with the nodes of the original knowledge graph. If a target node (evaluation index / technical entity term) already exists in the original graph, then the node is reused, and only its node attribute information is updated (integrating the target document attribute information corresponding to the document to be evaluated and supplementing new attribute content).

[0108] If a target node does not exist in the original map, a new node is created, a unique node identifier is assigned to it, the node type is labeled, and the corresponding target document attribute information is associated as the attribute information of the node.

[0109] Simultaneously, based on the target document relationship information (the correlation between technical means and technical problems) of the documents to be evaluated, and the association logic between the target node and the original graph nodes, graph edges can be added or associated: If the target node is a newly added node, a new graph edge is added to connect the newly added node with the associated nodes in the original graph, clarifying the semantic association type of the edge and realizing the logical connection between the newly added node and the original graph.

[0110] If the target node is a reused node, then based on the target document relationship information, supplement the associated edges between this node and other nodes (if they do not exist), or strengthen the label association of existing associated edges.

[0111] Addition of label association: For all newly added / updated nodes and graph edges, bind the unique identifier of the literature to be evaluated (such as literature number, DOI number), so that these nodes / edges are associated with the unique label of the literature to be evaluated, which is consistent with the label association rules of the original graph and ensures traceability.

[0112] Graph Verification and Integration: The updated graph is verified for completeness and relevance, and issues such as node redundancy and edge association errors are corrected. The original graph information is integrated with the newly added information to form a complete and accurately correlated target knowledge graph, which is then stored in a graph database.

[0113] Step 503: Based on the target graph nodes in the target knowledge graph that are connected to the graph nodes corresponding to the document to be evaluated, the node attribute information corresponding to the target graph nodes, and the target association relationship, the target graph nodes, the node attribute information corresponding to the target graph nodes, and the target association relationship are used as the third information associated with the document to be evaluated.

[0114] The graph node corresponding to the document to be evaluated refers to the graph node that is uniquely associated with the document to be evaluated (i.e., the node contributed by the document to be evaluated) by matching or adding the target document evaluation indicators and target technical entity words in the target knowledge graph with the target document evaluation indicators and target technical entity words as core elements.

[0115] Target knowledge graph nodes refer to all knowledge graph nodes in the target knowledge graph that have a direct or indirect connection with the "knowledge graph node corresponding to the document to be evaluated", including existing related nodes in the original knowledge graph and newly added related nodes after the update.

[0116] Target association refers to the graph edges in the target knowledge graph that connect the graph nodes corresponding to the literature to be evaluated to the target graph nodes, representing the technical association logic between the two types of nodes (such as the application association between indicators and entities, the correspondence between technical means and problems, etc.).

[0117] After obtaining the updated target knowledge graph, the target graph nodes connected to the corresponding graph nodes of the document to be evaluated, the node attribute information of the target graph nodes, and the target association relationships can be used as the third information associated with the document to be evaluated. Specifically, based on the unique identifier of the document to be evaluated, all graph nodes bound to that document (i.e., the nodes added / reused in step 502 that are associated with the document to be evaluated) can be retrieved and located in the target knowledge graph as the retrieval core. Using the located core node as a benchmark, all graph nodes in the target knowledge graph that have a direct or indirect connection with the core node are retrieved, which are the target graph nodes. Nodes without substantial technical relevance are excluded during the screening process to ensure that the target graph nodes are highly relevant to the technical content of the document to be evaluated. Then, the node attribute information (including existing attributes of the original graph and the attributes added in step 502) corresponding to each target graph node can be extracted one by one, as well as the target association relationship connecting the core node and the target graph node (clarifying the semantic type of each edge), ensuring that there is no missing or redundant information. Finally, the extracted target graph nodes, node attribute information, and target association relationships are structured and organized according to the logic of "core node - target node - association relationship - node attribute", and the corresponding literature tracing identifiers (identifiers of the literature to be evaluated and the related published literature) are marked for each piece of information, forming standardized third information.

[0118] In this embodiment, the third information includes: the original graph nodes of the target knowledge graph before the update, the node attribute information and original associations of the original graph nodes, the newly added graph nodes in the target knowledge graph, the node attribute information and newly added associations of the newly added graph nodes, and the associated nodes in the original knowledge graph that are connected to the graph nodes corresponding to the document to be evaluated, the node attribute information and corresponding associations of the associated nodes.

[0119] In practical applications, the information related to the literature to be evaluated introduced into the updated knowledge graph can include newly added nodes and new relationships, which can serve as an important basis for literature evaluation.

[0120] This application's embodiments extract standardized target information from the literature to be evaluated, and then perform precise incremental updates to the original target domain knowledge graph. This preserves the integrity of the original domain's technical knowledge while supplementing it with relevant technical elements of the literature to be evaluated, ensuring the timeliness and comprehensiveness of the knowledge graph. Simultaneously, third-party information highly correlated with the literature to be evaluated is selected from the updated target knowledge graph, achieving precise matching between the literature to be evaluated and the domain's structured knowledge. This provides reliable and comprehensive knowledge support for subsequent multi-dimensional evaluation of the literature, effectively improving the accuracy and efficiency of literature evaluation, and simultaneously realizing deep linkage between the dynamic iteration of the knowledge graph and literature evaluation.

[0121] Next, combined Figure 6 The process of screening high-value literature is described in detail.

[0122] Reference Figure 6 The diagram illustrates a flowchart of the steps involved in a target document screening method provided in an embodiment of this application. Figure 6 As shown, the target literature screening method may include steps 601 to 603.

[0123] Step 601: Obtain third-party literature that matches the target domain and the evaluation dimension, and the literature evaluation results of the third-party literature.

[0124] In this embodiment, the third document refers to a reference document that belongs to the same target domain as the document to be evaluated and fits the same evaluation dimension. It is the core reference object for value scoring comparison.

[0125] The evaluation results of third-party literature refer to the qualitative / quantitative evaluations and corresponding bases of the third-party literature in the specified evaluation dimensions using the same evaluation system (the same large language model) as the literature to be evaluated. The evaluation results of the third-party literature are consistent with the evaluation results of the literature to be evaluated, and provide support for the value scoring.

[0126] After obtaining the evaluation results of the literature to be evaluated, third-party literature matching the target field and evaluation dimensions can be acquired, along with the literature evaluation results of the third-party literature. Specifically, using target field terms and evaluation dimension terms as core search terms, multi-source literature databases (academic paper databases, patent databases) can be accessed to retrieve published literature within the field that meets both conditions. Text classification algorithms are used to verify the degree of fit between the literature topic and the target field and evaluation dimensions, eliminating off-topic, low-quality, and duplicate literature to form a set of third-party literature.

[0127] If the third-party document has already been evaluated by the large language model (e.g., included in the full set of documents to be compared in the previous section), the corresponding evaluation results can be retrieved directly. If it has not been evaluated, each third-party document is input into the same large language model used to evaluate the documents to be evaluated, and the model prompt words constructed in the previous section are input simultaneously to ensure that the evaluation criteria are consistent, and the literature evaluation results (including qualitative conclusions and basis) for each third-party document are generated.

[0128] Step 602: Determine the first value score of the third document based on the document evaluation results, and determine the second value score of the document to be evaluated based on the evaluation results.

[0129] The first value score refers to the score (e.g., 0-10 points, 0%-100%) converted according to unified quantitative rules based on the literature evaluation results of third-party literature. It reflects the core value level of the third-party literature in the target field and evaluation dimensions and is used to compare with the value score of the literature to be evaluated.

[0130] The second value score refers to the score converted from the literature evaluation results of the literature to be evaluated using the same quantitative rules as the first value score, ensuring the fairness and objectivity of the comparison between the two.

[0131] After obtaining the third-party literature, a first value score can be determined based on the literature evaluation results, and a second value score can be determined based on the evaluation results. Specifically, a pre-set scoring conversion system can be retrieved to quantify the qualitative descriptions of the evaluation results (such as "excellent," "good," and "average") into specific scores, or weights can be allocated based on the core quantitative indicators in the evaluation results (such as innovation, industrialization potential, and performance) (e.g., academic dimensions: innovation 40%, citation potential 30%, methodological feasibility 30%), forming a unified scoring rule.

[0132] For the calculation of the first value score: For each third document, its evaluation results are broken down according to the above rules, the scores of each core indicator are quantified and weighted and summarized to obtain the first value score of a single third document. If there are multiple third documents, the average score can be calculated as the overall reference score, or the individual scores can be retained for comparison.

[0133] For the calculation of the second value score: the same quantitative rules and weight allocation as the first value score are used to decompose the evaluation results of the literature to be evaluated, and the second value score is obtained by quantitative calculation to ensure that the comparison benchmark of the two scores is consistent.

[0134] Step 603: Based on the first value score and the second value score, select the target document with the highest value score from the third document and the document to be evaluated.

[0135] The target literature is selected by comparing the first value score with the second value score. The literature with the highest score (the third literature or the literature to be evaluated) is the optimal value literature in the corresponding dimension in this evaluation process.

[0136] After obtaining the first and second value scores, the target document with the highest value score can be selected from the third document and the document to be evaluated based on the first and second value scores. That is, the document with the highest score is determined as the target document—if the second value score is higher than all (or the average) first value scores, the document to be evaluated is the target document; if a third document has the highest first value score, that third document is the target document.

[0137] This application's embodiments provide a reasonable value benchmark for the literature to be evaluated by accurately acquiring third-party literature that matches both the target field and the evaluation dimensions, thus avoiding the subjectivity of evaluating a single literature. Relying on unified quantitative rules, the evaluation results are transformed into comparable value scores, ensuring fairness in the comparison. Through score comparison and tie-breaking mechanisms, the optimal value literature is quickly identified, forming a complete logical chain of "obtaining a reference - quantifying the score - selecting the optimal," which not only connects to the aforementioned large language model evaluation system but also improves the objectivity and accuracy of literature value judgment, providing users with clear decision-making basis.

[0138] Reference Figure 7 The diagram shows a schematic representation of a document evaluation device provided in an embodiment of this application. Figure 7 As shown, the document evaluation device 700 may include the following modules: The acquisition module 710 is used to acquire the target domain, evaluation dimensions, and literature to be evaluated input by the user. The first information retrieval module 720 is used to retrieve first information from the Internet that belongs to the target domain and corresponds to the evaluation dimension within a target time period. The second information retrieval module 730 is used to retrieve second information associated with the document to be evaluated from the document database; The third information acquisition module 740 is used to acquire third information associated with the document to be evaluated from a pre-constructed knowledge graph corresponding to the target domain. The prompt word construction module 750 is used to construct model prompt words based on the first information, the second information, the third information, the target domain, and the evaluation dimension; The literature evaluation module 760 is used to input the literature to be evaluated and the model prompt words into a large language model for literature evaluation, so that the large language model outputs the evaluation result of the literature to be evaluated in the evaluation dimension and the evaluation basis information corresponding to the evaluation result.

[0139] Optionally, the first information retrieval module includes: The target time acquisition unit is used to acquire the target time of the most recent update of the knowledge graph. A target time period determination unit is used to determine the time period between the target time and the current time as the target time period; The information retrieval unit is used to retrieve from the Internet literature and technical description information that belongs to the target field and corresponds to the evaluation dimension within the target time period; The first information determining unit is used to determine the document and the technical description information as the first information.

[0140] Optionally, the second information retrieval module includes: The first document extraction unit is used to extract the first document information from the document to be evaluated; wherein the first document information includes at least: the document abstract, research purpose, methods used to solve the problem, evaluation indicators and evaluation set used in the research, and research conclusions of the document to be evaluated; The second document acquisition unit is used to acquire the second document information of each document in the document database; The second information acquisition unit is used to filter out the second information from the second document information whose comprehensive similarity is greater than a similarity threshold based on the comprehensive similarity between the first document information and the second document information.

[0141] Optionally, the device further includes: The document acquisition module is used to acquire published documents belonging to a specified field; wherein, the specified field includes at least the target field; The document information acquisition module is used to acquire the document evaluation indicators, technical entity terms, document relationship information, and document attribute information associated with the document evaluation indicators and technical entity terms of the published documents; wherein, the document relationship information includes at least: information on the technical means used and the technical problems solved by the published documents, and the document attribute information is structured information associated with the document evaluation indicators and technical entity terms; The knowledge graph construction module is used to construct a knowledge graph for the specified domain by using the literature evaluation indicators and the technical entity words as graph nodes, associating the literature attribute information with each graph node as the node attribute information of the corresponding graph node, and uniquely labeling and associating each graph node and each graph edge with the corresponding published literature.

[0142] Optionally, the third information acquisition module includes: The document information acquisition unit is used to acquire the target document evaluation index, target technical entity words, target document relationship information, and target document attribute information associated with the target document evaluation index and the target technical entity words of the document to be evaluated. The knowledge graph updating unit is used to update the knowledge graph corresponding to the target domain according to the target document evaluation index, the target technical entity words, the target document relationship information and the target document attribute information, so as to obtain the target knowledge graph; The third information acquisition unit is used to obtain the target graph node that is connected to the graph node corresponding to the document to be evaluated in the target knowledge graph, the node attribute information corresponding to the target graph node, and the target association relationship, and to use the target graph node, the node attribute information corresponding to the target graph node, and the target association relationship as the third information associated with the document to be evaluated.

[0143] Optionally, the third information includes: the original graph nodes of the target knowledge graph before the update, the node attribute information and original associations corresponding to the original graph nodes, the newly added graph nodes in the target knowledge graph, the node attribute information and newly added associations corresponding to the newly added graph nodes, and the associated nodes in the original knowledge graph that are connected to the graph nodes corresponding to the document to be evaluated, the node attribute information and corresponding associations corresponding to the associated nodes.

[0144] Optionally, the device further includes: The evaluation result acquisition module is used to acquire third-party literature that matches the target field and the evaluation dimension, and the literature evaluation results of the third-party literature; The value scoring determination module is used to determine a first value score of the third document based on the document evaluation results, and to determine a second value score of the document to be evaluated based on the evaluation results. The target document screening module is used to screen the target document with the highest value score from the third document and the document to be evaluated based on the first value score and the second value score.

[0145] This application embodiment integrates first information from internet retrieval, second information from literature database association, and third information from the target domain knowledge graph. By combining the target domain and evaluation dimensions, it accurately constructs model prompt words. After inputting the literature to be evaluated and the prompt words into the large language model, the multi-source information provides a comprehensive and professional basis for the evaluation. At the same time, the prompt words clarify the core dimensions and domain logic of the evaluation, making the evaluation results output by the large language model more realistic and traceable, thereby effectively improving the accuracy and interpretability of literature evaluation.

[0146] This application also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the above-described literature evaluation method.

[0147] Figure 8 A schematic diagram of the structure of an electronic device 800 according to an embodiment of this application is shown. Figure 8 As shown, the electronic device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. The RAM 803 can also store various programs and data required for the operation of the electronic device 800. The CPU 801, ROM 802, and RAM 803 are interconnected via bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0148] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, microphone, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0149] The various processes and handling described above can be executed by processing unit 801. For example, the methods of any of the above embodiments can be implemented as computer software programs tangibly contained in a computer-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more actions of the methods described above can be performed.

[0150] Additionally, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned document evaluation method.

[0151] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes said element.

[0152] The present application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present application. Therefore, the content of this specification should not be construed as a limitation of the present application.

Claims

1. A literature evaluation method, characterized in that, The method includes: Obtain the target domain, evaluation dimensions, and literature to be evaluated as input by the user; Retrieve first information from the internet that belongs to the target domain and corresponds to the evaluation dimension within the target time period; Second information related to the document to be evaluated is retrieved from the literature database; Obtain third information associated with the document to be evaluated from the pre-constructed knowledge graph corresponding to the target domain; Based on the first information, the second information, the third information, the target domain, and the evaluation dimension, model prompt words are constructed. The document to be evaluated and the model prompts are input into a large language model for document evaluation, so that the large language model outputs the evaluation results of the document to be evaluated in the evaluation dimension and the evaluation basis information corresponding to the evaluation results.

2. The method according to claim 1, characterized in that, The step of retrieving first information from the internet within a target time period that belongs to the target domain and corresponds to the evaluation dimension includes: Obtain the target time of the most recent update of the knowledge graph; The time period between the target time and the current time is defined as the target time period; Retrieve from the Internet literature and technical description information that belong to the target field and correspond to the evaluation dimension within the target time period; The document and the technical description information are identified as the first information.

3. The method according to claim 1, characterized in that, The second information retrieved from the literature database that is associated with the document to be evaluated includes: Extract the first literature information from the literature to be evaluated; wherein the first literature information includes at least: the literature abstract, research purpose, methods used to solve the problem, evaluation indicators and evaluation set used in the research, and research conclusions of the literature to be evaluated; Obtain the second literature information of each document in the literature database; Based on the comprehensive similarity between the first document information and the second document information, the second information with a comprehensive similarity greater than the similarity threshold is selected from the second document information.

4. The method according to claim 1, characterized in that, Before obtaining the third information associated with the document to be evaluated from the pre-constructed knowledge graph corresponding to the target domain, the method further includes: Obtain published literature belonging to a specified field; wherein, the specified field includes at least the target field; The document evaluation indicators, technical entity terms, document relationship information, and document attribute information associated with the published documents are obtained; wherein, the document relationship information includes at least: information on the technical means used and the technical problems solved by the published documents, and the document attribute information is structured information associated with the document evaluation indicators and the technical entity terms; Using the literature evaluation indicators and the technical entity terms as graph nodes of the knowledge graph, the literature attribute information is associated with each graph node as the node attribute information of the corresponding graph node. Each graph node and each graph edge is uniquely labeled and associated with the corresponding published literature to construct the knowledge graph of the specified field.

5. The method according to claim 1, characterized in that, The step of obtaining third information associated with the document to be evaluated from a pre-constructed knowledge graph corresponding to the target domain includes: Obtain the target document evaluation indicators, target technical entity words, target document relationship information, and target document attribute information associated with the target document evaluation indicators and target technical entity words; Based on the target document evaluation indicators, the target technical entity words, the target document relationship information, and the target document attribute information, update the knowledge graph corresponding to the target domain to obtain the target knowledge graph; Based on the target graph nodes that are connected to the graph nodes corresponding to the document to be evaluated in the target knowledge graph, the node attribute information corresponding to the target graph nodes, and the target association relationship, the target graph nodes, the node attribute information corresponding to the target graph nodes, and the target association relationship are used as the third information associated with the document to be evaluated.

6. The method according to claim 5, characterized in that, The third information includes: the original graph nodes of the target knowledge graph before the update, the node attribute information and original associations of the original graph nodes, the newly added graph nodes in the target knowledge graph, the node attribute information and newly added associations of the newly added graph nodes, and the associated nodes in the original knowledge graph that are connected to the graph nodes corresponding to the document to be evaluated, the node attribute information and corresponding associations of the associated nodes.

7. The method according to claim 1, characterized in that, After the step of causing the large language model to output the evaluation results of the document to be evaluated in the evaluation dimension and the evaluation basis information corresponding to the evaluation results, the method further includes: Obtain third-party literature that matches the target domain and the evaluation dimensions, and obtain the literature evaluation results of the third-party literature; Based on the literature evaluation results, a first value score is determined for the third literature, and a second value score is determined for the literature to be evaluated based on the evaluation results. Based on the first value score and the second value score, the target document with the highest value score is selected from the third document and the document to be evaluated.

8. A document evaluation device, characterized in that, The device includes: The acquisition module is used to acquire the target domain, evaluation dimensions, and literature to be evaluated, as input by the user. The first information retrieval module is used to retrieve first information from the Internet that belongs to the target domain and corresponds to the evaluation dimension within the target time period. The second information retrieval module is used to retrieve second information associated with the document to be evaluated from the document database; The third information acquisition module is used to acquire third information associated with the document to be evaluated from the pre-constructed knowledge graph corresponding to the target domain. The prompt word construction module is used to construct model prompt words based on the first information, the second information, the third information, the target domain, and the evaluation dimension; The document evaluation module is used to input the document to be evaluated and the model prompt words into a large language model for document evaluation, so that the large language model outputs the evaluation result of the document to be evaluated in the evaluation dimension and the evaluation basis information corresponding to the evaluation result.

9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the document evaluation method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the document evaluation method according to any one of claims 1 to 7.

Images