General scientific research entity corpus construction method and device based on scientific literature, and medium

Abstract

The application discloses a general scientific research entity corpus construction method and device based on scientific literature and a medium, relates to the technical field of natural language processing and knowledge engineering, and comprises the following steps: building an extensible model pool; automatically pre-annotating a scientific literature abstract set for general scientific research entities to obtain an extracted general scientific research entity set and a confidence set; filtering and screening the confidence set to obtain a difficult sample set; extracting the difficult sample set to obtain a large language model pre-annotation result set; and obtaining a large language model correction annotation result set and a large language model correction degree set. The application solves the technical problems that the existing general scientific research corpus construction process is imperfect, the model lacks an effective feedback optimization mechanism, and the scientific research entity extraction efficiency is low and the accuracy is insufficient, and achieves the technical effects of realizing a high-quality general scientific research entity corpus construction process and improving the entity extraction efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and knowledge engineering, specifically to a method, device, and medium for constructing a general scientific research entity corpus based on scientific and technological literature. Background Technology

[0002] In scientific research and knowledge management, the rapid and accurate extraction of structured knowledge from massive amounts of scientific and technological literature is a crucial requirement. Current mainstream methods for constructing such entity corpora have significant shortcomings: while purely manual annotation offers high accuracy, it is extremely inefficient, costly, and difficult to apply on a large scale; automatic extraction based on rules or traditional machine learning models, while improving automation, suffers from high domain dependence, fixed entity types, and still requires manually annotated corpora to support initial model training. In recent years, large language models have shown potential in information extraction tasks, but they lack accuracy in extracting specialized scientific entities, easily generating inaccurate or fictitious content; simultaneously, existing corpus construction processes are mostly one-way, lacking effective feedback mechanisms, and cannot utilize manually verified information for continuous optimization of the extraction model, leading to a bottleneck in the efficiency and quality improvement of corpus construction. Summary of the Invention

[0003] This application provides a method, device, and medium for constructing a general scientific research entity corpus based on scientific and technological literature, which addresses the technical problems of imperfect general scientific research corpus construction processes and the lack of effective feedback optimization mechanisms in existing technologies, resulting in low efficiency and insufficient accuracy in extracting scientific research entities.

[0004] In view of the above problems, this application provides a method, equipment and medium for constructing a general scientific research entity corpus based on scientific and technological literature.

[0005] The first aspect of this application provides a method for constructing a general scientific research entity corpus based on scientific and technological literature, the method comprising:

[0006] An extensible model pool is constructed, comprising a dedicated model set and a large language model interface set. The dedicated model set is used to automatically pre-annotate a set of scientific literature abstracts for general research entities, resulting in an extracted set of general research entities and a confidence set. The general research entities include at least one of research questions, research methods, algorithm models, software systems, metrics, instruments and equipment, theoretical principles, and data. The confidence set is filtered according to a preset confidence threshold, and based on the filtering results, the extracted set of general research entities and the scientific literature abstracts are partitioned into sample mappings, resulting in a set of difficult samples and a set of general research entities extracted by the dedicated model. The large language model interface set is used to extract from the difficult samples according to a preset prompt word template, resulting in an extraction result set. The extracted result set is then fused with the extracted set of general research entities to obtain a large language model pre-annotation result set. The data is then processed through a corpus. Processing personnel review and correct the pre-annotated result set of the large language model, record operations to obtain an operation record set and an un-operated large language model pre-annotated result set, and identify the degree of correction based on the operation record set to obtain a large language model corrected annotation result set and a large language model correction degree set. Corpus processing personnel review and correct the general scientific entity extraction set of the dedicated model, and extract and identify correction operation records to obtain a dedicated model un-corrected extraction set and a dedicated model corrected extraction set of general scientific entities. The large language model corrected annotation result set, the un-operated large language model pre-annotated result set, the dedicated model un-corrected extraction set of general scientific entities, and the dedicated model corrected extraction set of general scientific entities are stored as a general scientific entity corpus. The large language model correction degree set, the large language model corrected annotation result set, and the corresponding large language model pre-annotated result set are associated and stored as feedback data, and the feedback data is used to optimize the scalable model pool.

[0007] In a second aspect, this application provides an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the method for constructing a general scientific entity corpus based on scientific and technological literature provided in this application.

[0008] In a third aspect of this application, a computer-readable storage medium is provided, storing a computer program for executing the method for constructing a general scientific research entity corpus based on scientific and technological literature provided in this application.

[0009] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0010] A scalable model pool is built; a dedicated model set is used to automatically pre-annotate a set of scientific literature abstracts with general research entities, resulting in an extracted set of general research entities and a confidence set; the confidence set is filtered according to a preset confidence threshold, and based on the filtering results, the extracted set of general research entities and the scientific literature abstracts are mapped and partitioned to obtain a set of difficult samples and a set of general research entities extracted by the dedicated model; the large language model interface set is used to extract from the difficult sample set according to a preset prompt word template, resulting in an extraction result set; the extracted result set is then fused with the extracted set of general research entities to obtain a large language model pre-annotation result set; the large language model pre-annotation result set is reviewed and corrected by corpus processing personnel, and the operations are recorded to obtain an operation record set and a set of non-operation large language models. The system pre-annotates the language model results set and identifies the degree of correction based on the operation record set, obtaining a large language model correction annotation result set and a large language model correction degree set. The system then reviews and corrects the general scientific entity set extracted by the dedicated model through corpus processing personnel, extracting and identifying correction operation records to obtain a dedicated model uncorrected general scientific entity set and a dedicated model corrected general scientific entity set. These sets—the large language model correction annotation result set, the uncorrected large language model pre-annotated result set, the dedicated model uncorrected general scientific entity set, and the dedicated model corrected general scientific entity set—are stored as a general scientific entity corpus. The large language model correction degree set, the large language model correction annotation result set, and the corresponding large language model pre-annotated result set are associated and stored as feedback data. This feedback data is used to optimize the scalable model pool. This process achieves a high-quality general scientific entity corpus construction workflow, improving the technical effect of entity extraction efficiency and accuracy. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating the method for constructing a general scientific research entity corpus based on scientific and technological literature, provided in the embodiments of this application;

[0013] Figure 2 This is a schematic diagram of the structure of an electronic device provided in this application.

[0014] Explanation of reference numerals in the attached drawings: Processor 21, Memory 22, Input device 23, Output device 24. Detailed Implementation

[0015] This application provides a method, device, and medium for constructing a general scientific research entity corpus based on scientific and technological literature. This addresses the technical problems in existing technologies, such as imperfect general scientific research corpus construction processes and the lack of effective feedback optimization mechanisms in models, which lead to low efficiency and insufficient accuracy in extracting scientific research entities.

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0017] Example 1, as Figure 1 As shown, this application provides a method for constructing a general scientific research entity corpus based on scientific and technological literature, the method comprising:

[0018] Step S100: Build an extensible model pool, wherein the extensible model pool includes a dedicated model set and a large language model interface set.

[0019] Specifically, the scalable model is built, comprising two core modules: a dedicated model set and a large language model interface set. The dedicated model set consists of named entity recognition models based on pre-trained architectures such as BERT and RoBERTa, further pre-trained on large-scale scientific corpora such as S2ORC, and then fine-tuned with high-quality manually annotated general scientific entity data. These models are specifically optimized for scientific entity recognition tasks, such as the recognition of research questions and algorithm models, and have the advantages of fast inference speed, low operating cost, and high professionalism. The large language model interface set is configured with API access permissions for mainstream large language models such as GPT, DeepSeek, and GLM. Relying on the powerful generalization ability of large models, it can handle extraction scenarios that are difficult for dedicated models to process, such as complex sentence structures and new concept entities. The two types of models work together to form a scalable model pool.

[0020] Step S200: Call the dedicated model set to automatically pre-label the scientific literature abstract set with general scientific entities, and obtain the extracted general scientific entity set and confidence set, wherein the general scientific entity includes at least one of research questions, research methods, algorithm models, software systems, metrics, instruments and equipment, theoretical principles and data.

[0021] Specifically, a dedicated model set is called from a scalable model pool. This dedicated model set is based on pre-trained architectures such as BERT and RoBERTa, further pre-trained on large-scale scientific corpora such as S2ORC, and fine-tuned with high-quality manually annotated general scientific entity data to obtain named entity recognition models with specialized recognition capabilities for scientific research scenarios. Subsequently, this dedicated model is used to batch process the imported collection of scientific literature abstracts, automatically identifying and extracting general scientific entities, covering at least one of research questions, research methods, algorithm models, software systems, metrics, instruments and equipment, theoretical principles, and data, forming an extracted set of general scientific entities. At the same time, according to the defined confidence calculation rules, the dedicated model will identify the probability of all tokens constituting each extracted general scientific entity on its predicted label, and perform mean processing on these probabilities, such as geometric mean or arithmetic mean calculation, to obtain the confidence score corresponding to each entity. Finally, the scores are summarized to form a confidence set that corresponds one-to-one with the extracted set of general scientific entities.

[0022] Step S300: Filter the confidence set according to the preset confidence threshold, and perform sample mapping division on the extracted general scientific research entity set and the scientific literature abstract set based on the filtering results to obtain the difficult sample set and the general scientific research entity set extracted by the special model.

[0023] Specifically, a pre-set confidence threshold, such as 0.75, is loaded and can be flexibly adjusted according to the entity extraction accuracy requirements of different research fields. The generated confidence set is then filtered, and all confidence scores below the threshold are included in the filtering results. The entities corresponding to these scores represent content that is difficult for dedicated models to accurately identify. At the same time, complete abstracts from the collection of scientific literature abstracts that were not identified by the dedicated model, including types such as research questions, research methods, and algorithm models, are also added to the filtering results. Next, based on the filtering results, sample mapping is carried out: on the one hand, the abstract fragments of the literature containing entities with insufficient confidence and the complete abstracts of entities that were not identified are integrated to form a set of difficult samples. These samples need to be further extracted by a large language model. On the other hand, the extracted general research entities corresponding to confidence scores greater than or equal to the pre-set confidence threshold in the confidence set are directly classified into the set of general research entities extracted by the dedicated model. This set only needs to be manually reviewed and does not require secondary processing by the large model. This division ensures the efficient use of highly reliable results and clarifies the scope for accurately tackling difficult samples.

[0024] Step S400: Call the large language model interface set to extract the difficult sample set according to the preset prompt word template, obtain the extraction result set, and fuse the extracted general scientific research entity set based on the extraction result set to obtain the large language model pre-annotation result set.

[0025] Specifically, a large language model interface set, including APIs for GPT, DeepSeek, and GLM, is called from a scalable model pool. Specialized extraction is performed on the obtained set of challenging samples, following a pre-defined prompt template. This template specifies the types of research entities, requires precise extraction, avoids ambiguous content, and outputs in strict JSON format. This ensures the large model clearly understands the extraction task boundaries. The challenging sample text and extraction requirements are passed to the large model, guiding it to identify and output general research entities such as research questions, methods, and algorithms, forming an extraction result set. Subsequently, a result fusion process is initiated, integrating the extraction result set output by the large language model with the general research entity set generated by the specialized model. Only high-confidence entities with a confidence level higher than a pre-set confidence threshold are retained. Duplicate entities are removed using a deduplication algorithm, and entities missed by the specialized model are supplemented. Finally, a pre-annotated result set from the large language model, balancing high confidence and comprehensive coverage, is formed, providing basic annotated text for subsequent manual review.

[0026] Step S500: The corpus processing personnel review and correct the pre-annotated result set of the large language model, record the operations to obtain the operation record set and the pre-annotated result set of the large language model without operations, and identify the degree of correction based on the operation record set to obtain the corrected annotation result set and the correction degree set of the large language model.

[0027] Specifically, corpus processing personnel obtain the generated large language model pre-annotation result set through the system's web interface, and review each general scientific research entity, research question, and research method within it. The focus is on verifying the accuracy of entity boundaries, such as whether there are any missing or redundant entity segments, the correctness of entity types, whether the algorithm model is mislabeled as a research method, and the completeness of entities, such as whether there are any missing or unannotated entities. During the review process, all operations performed by the processing personnel, such as modifying entity boundaries, modifying entity types, adding entities, and deleting entities, are recorded and stored in real time by the system, forming an operation record set. If a pre-annotation result does not require correction, there is no corresponding correction operation record. Entries in the large language model pre-annotation result set that pass the review directly without any correction are classified as the un-operated large language model pre-annotation result set. Subsequently, the degree of correction is identified based on the set of operation records: according to preset operation weights, such as 1 point for modifying an entity, 1.5 points for adding an entity, and 0.5 points for deleting an entity, the total score of the correction operation weight for each pre-annotated result that needs correction is calculated; then, the total number of all entities before the operation of that pre-annotated result is determined, that is, the number of entities output during the model pre-annotation stage, and the correction score is calculated as: correction score = total score of correction operation weight / total number of all entities in the text before the operation; then, according to preset correction degree classification rules, a score < 0.5 is low correction degree, 0.5 ≤ score < 1.5 is medium correction degree, and a score ≥ 1.5 is high correction degree, the correction scores are classified into levels, and finally, the accurately annotated content after manual correction is obtained, that is, the set of corrected annotation results of the large language model and the correction level data of each corpus, that is, the set of correction degree of the large language model. At the same time, the annotation results after manual correction are integrated into the set of corrected annotation results of the large language model, and finally, the set of operation records, the set of pre-annotated results of the large language model without operation, the set of corrected annotation results of the large language model, and the set of correction degree of the large language model are obtained simultaneously.

[0028] Step S600: The corpus processing personnel review and correct the general scientific research entity set extracted by the special model, and extract and identify the correction operation records to obtain the general scientific research entity set extracted by the special model without correction and the general scientific research entity set extracted by the special model with correction.

[0029] Specifically, the corpus processing personnel conduct a sample review of the general scientific research entity set extracted by the dedicated model. This review focuses on verifying the accuracy of entity boundaries, the rationality of entity type classification, and whether any entities were omitted or incorrectly extracted. For any issues discovered during the review, targeted corrective actions are taken, including adjusting entity boundaries, correcting entity types, adding omitted entities, and deleting incorrectly extracted entities. Simultaneously, the details of each corrective action are extracted and recorded in real time to form a correction operation record. Based on the review results and correction operation records, entities without issues and requiring no correction are classified as the uncorrected general scientific research entity set extracted by the dedicated model, while entities that conform to the standards after correction are classified as the corrected general scientific research entity set extracted by the dedicated model. This provides accurate and compliant basic data for the subsequent integration and storage of the general scientific research entity corpus.

[0030] Step S700: Store the set of large language model correction annotation results, the set of large language model pre-annotation results without operation, the set of general scientific research entities extracted by the special model without correction, and the set of general scientific research entities extracted by the special model with correction as a general scientific research entity corpus. Link and store the set of large language model correction degree, the set of large language model correction annotation results, and the corresponding set of large language model pre-annotation results as feedback data. Optimize the scalable model pool using the feedback data.

[0031] Specifically, the large language model's corrected annotation result set, the uncorrected large language model's pre-annotated result set, and the dedicated model's uncorrected and corrected general scientific research entity sets are integrated and archived into a high-quality general scientific research entity corpus. This corpus is then stored in the system database to provide structured data support for subsequent scientific knowledge retrieval and analysis tasks. Simultaneously, following the principle of associating corrected data with original pre-annotated data, the obtained large language model correction degree set (including high, medium, and low correction levels), the corresponding large language model corrected annotation result set, and the pre-annotated result set before correction are bound and stored to form feedback data. This data directly reflects the shortcomings and optimization directions of the knowledge extraction model.

[0032] Subsequently, the feedback data was used to conduct targeted optimizations on the scalable model pool, which includes a dedicated model set and a large language model interface set. On the one hand, samples with a "high" correction level and their correct annotations were selected from the feedback data, i.e., the large language model correction annotation result set, as key training data. This data was used to incrementally train or fine-tune the dedicated model set, forcing the model to focus on sample types that had previously performed poorly in extraction, such as easily confused algorithm models and research method entities, thereby improving the accuracy of the dedicated model. On the other hand, the error patterns of high-correction-level samples in the large language model extraction were analyzed, such as entity boundary segmentation deviation and type misjudgment. Based on these patterns, the preset prompt word templates were optimized, such as adding examples of easily confused entities to the prompt words and clarifying entity boundary annotation requirements, thereby improving the reliability of the large model extraction. In addition, combined with the strategy optimization logic, if the feedback data showed that the correction level of the large model was significantly lower than that of the dedicated model for a certain type of literature, the model calling strategy was dynamically adjusted to directly assign this type of literature to the large model for processing, thereby maximizing the overall efficiency of the scalable model pool and ultimately forming a closed loop of corpus storage-model optimization-better extraction.

[0033] In one possible implementation, step S200 further includes:

[0034] Step S210: Traverse the extracted general scientific research entity set for probability identification, and perform mean processing on the identification results to obtain a confidence set.

[0035] Step S220: Wherein, the probability identification is to identify the probability of all tokens constituting any general scientific research entity on its predicted label.

[0036] Specifically, the process begins by traversing the set of general scientific research entities extracted from the set of specialized models. This set includes general scientific research entities such as research questions, research methods, and algorithm models. For each general scientific research entity in the set, probabilistic identification is performed. This involves identifying the probability values ​​of all tokens constituting that entity on their predicted labels, such as entity type labels like "Research Question-B" or "Algorithm Model-I." These probability values ​​directly reflect the reliability of the specialized model's judgment on the entity type of a single token. Subsequently, the predicted probabilities of all tokens corresponding to each general scientific research entity are averaged using a mean algorithm, such as the geometric mean or arithmetic mean, to obtain the overall confidence score for each general scientific research entity. Finally, the confidence scores of all general scientific research entities are aggregated according to their correspondence to form a confidence set that matches the set of extracted general scientific research entities.

[0037] The core of probabilistic identification is to extract a set of general scientific research entities from the collection of scientific literature abstracts using a dedicated model set. For any one of these general scientific research entities, covering any one of the following categories: research question, research method, algorithm model, software system, metric, instrumentation, theoretical principle, and data, the entity is broken down and analyzed. This breaks down the individual general scientific research entity into all the tokens that constitute its textual semantics, such as Chinese characters, words, or sub-words. Then, the dedicated model calls its internal prediction probability output module to identify the probability value of each token on the prediction label assigned to it by the dedicated model, such as the initial label "Research Method-B" and the internal label "Research Method-I" for the entity type. This probability value directly reflects the reliability of the dedicated model's judgment on the entity type of a single token and is the basic data source for calculating the average confidence of the entity in subsequent steps.

[0038] In one possible implementation, step S300 further includes:

[0039] Step S310: Add the confidence scores in the confidence score set that are less than the preset confidence score threshold to the filtering results.

[0040] Step S320: Based on the filtering results, map the general scientific research entities extracted from the general scientific research entity set, and add the mapping results to the difficult sample set.

[0041] Step S330: Add the scientific and technological literature abstracts in the scientific and technological literature abstract set that have not been identified to the difficult sample set.

[0042] Step S340: Add the set of general scientific research entities corresponding to the confidence scores greater than or equal to the preset confidence threshold from the confidence score set to the set of general scientific research entities extracted by the special model.

[0043] Specifically, a pre-set confidence threshold, such as 0.75, is loaded. This threshold can be adjusted according to the entity extraction accuracy requirements of different research fields. The generated confidence sets are compared one by one, and all confidence scores below the threshold are filtered out and added to the filter results. These low confidence scores correspond to entities that are difficult for specialized models to accurately identify and are the focus of subsequent processing. In addition, for cases where the confidence score is greater than or equal to the pre-set confidence threshold, but the number of general research entities (including research questions, research methods, and algorithm models) marked in a single scientific literature abstract is less than three, even if the confidence requirement is met, their corresponding confidence entries are still added to the filter results. Through the superimposed screening of dual standards, low-quality extraction results with insufficient confidence are excluded, and the problem of missing information due to insufficient entity count is avoided, providing a more comprehensive screening basis for the accurate identification and secondary extraction of difficult samples.

[0044] Next, using the filtering results as an index, the extracted general scientific research entities that correspond one-to-one with the low confidence scores are located in the obtained general scientific research entity set. The complete sentences or paragraphs containing these entities are used as mapping results and added to the difficult sample set to ensure coverage of entity-related texts where the confidence of the dedicated model is insufficient.

[0045] Finally, a comprehensive review of the initially imported collection of scientific literature abstracts was conducted to identify abstract texts that did not have any general scientific research entities such as research questions, research methods, or algorithm models recognized by the dedicated model. These abstracts, which may have omissions of entities, were also added to the set of difficult samples. This resulted in a complete set of difficult samples that included low-confidence entity-related texts and texts without entity recognition, laying the foundation for subsequent secondary extraction using the large language model interface set.

[0046] The selection criteria were clearly defined by a pre-set reliability threshold, which was determined based on the extraction accuracy verification of the dedicated model on historical scientific literature corpora and was used to define the reliability of the entity extraction results. Subsequently, the confidence set output by the dedicated model was traversed, and all entries with confidence scores greater than or equal to the pre-set reliability threshold were selected. These entries correspond to general scientific entities, covering core scientific entity types such as research questions, research methods, software systems, metrics, and instruments. Confidence verification showed that their boundary accuracy and type correctness were at a high level, meeting the basic quality requirements for subsequent corpus construction. Finally, these high-confidence entities were integrated into a general scientific entity set extracted by the dedicated model. This set only requires simple review by corpus processing personnel and does not require secondary extraction using a large language model, which reduces unnecessary manual and model resource consumption and provides high-quality basic data for the general scientific entity corpus.

[0047] In one possible implementation, step S500 further includes:

[0048] Step S510: Each operation record in the operation record set includes at least one of modifying entity boundaries, modifying entity type, adding entity, and deleting entity.

[0049] Specifically, the operation log set is a key data record carrier for corpus processing personnel to review and correct the pre-annotation results. Each operation log focuses on the annotation optimization of general scientific research entities, namely research questions, research methods, algorithm models, software systems, metrics, instruments and equipment, theoretical principles, and data. It includes at least one of the following operation types: First, modifying entity boundaries, i.e., addressing cases where there are redundant or missing entity text fragments in the pre-annotation, such as mislabeling a deep learning-based image recognition algorithm as deep learning image recognition, adjusting the start and end positions of the entity to ensure accurate boundaries; Second, modifying entity types, i.e., correcting misjudgments of entity types in the pre-annotation, such as mislabeling a convolutional neural network algorithm model as a research method, rematching the correct entity type label; Third, adding entities, i.e., supplementing general scientific research entities that were missing in the pre-annotation, such as manually adding the entity and its corresponding type if the pre-annotation did not identify the Python data analysis software software entity in the abstract; Fourth, deleting entities, i.e. removing incorrectly labeled false entities in the pre-annotation, such as mislabeling a common description of a non-scientific entity as an instrument or equipment, deleting the incorrect label. These operation records are captured and stored by the system in real time, providing raw operation data support for subsequent calculations of correction levels, generation of large language model correction level sets, and optimization of scalable model pools.

[0050] In one possible implementation, step S500 further includes:

[0051] Step S520: According to the preset operation weight, traverse the operation record set to perform correction score identification and obtain the correction score set.

[0052] Step S530: Divide the correction score set according to the preset correction degree division rule to obtain the large language model correction degree set; wherein, the preset correction degree division rule is that when the score < 0.5, it is a low correction degree; when 0.5 ≤ score < 1.5, it is a medium correction degree; when the score ≥ 1.5, it is a high correction degree.

[0053] Specifically, preset operation weights are loaded, which are set according to the complexity and workload of different correction operations. For example, basic adjustment operations such as modifying entity boundaries and modifying entity types are assigned 1 point, adding new entities that require re-identification and annotation are assigned 1.5 points, and deleting erroneous entities are assigned 0.5 points. Then, the set of recorded operation records is traversed, and for each operation record, the total score of the correction operation weights for each pre-annotated result is calculated by accumulating the corresponding preset weights for operations such as modifying entity boundaries, modifying entity types, adding entities, and deleting entities. At the same time, the total number of entities in the scientific literature abstract text containing the pre-annotated result is obtained, and the correction score for each corpus is calculated using the formula: Correction Score = Total Correction Operation Weight Score / Total Number of Entities in the Text. Finally, the correction scores of all corpora are summarized according to their corresponding relationships to form a correction score set covering all pre-annotated results, providing a quantitative basis for subsequent steps to classify the degree of correction and generate a large language model correction degree set.

[0054] The system loads a pre-defined correction level classification rule, which clarifies the correspondence between correction scores and correction levels. When the correction score of a corpus in the generated correction score set is <0.5, the pre-annotation result corresponding to that corpus is considered to be of high quality and requires only minor adjustments, thus being classified as low correction level. When the correction score satisfies 0.5 ≤ score < 1.5, the pre-annotation result is considered to be basically usable but requires a certain amount of modification, thus being classified as medium correction level. When the correction score is ≥1.5, the pre-annotation result is considered to be of poor quality and requires a large amount of re-annotation or almost complete re-annotation, thus being classified as high correction level. Subsequently, the system iterates through each score in the correction score set, matching the corresponding correction level one by one according to the above rule. Finally, the correction level of all corpora is summarized to form a large language model correction level set that corresponds one-to-one with the large language model correction annotation result set.

[0055] In one possible implementation, step S600 further includes:

[0056] Step S610: Extract from the feedback data the large language model correction annotation results and the corresponding large language model pre-annotation result set that are associated with high correction degree in the large language model correction degree set as training data, and perform incremental training on the dedicated model set.

[0057] Step S620: Extract from the feedback data the large language model correction annotation results and the corresponding large language model pre-annotation result set that are associated with high correction degree in the large language model correction degree set as pattern correction data. Optimize the preset prompt word template based on the pattern correction data to obtain the optimized prompt word template.

[0058] Specifically, the process begins with targeted filtering of the associated feedback data. This feedback data includes a set of large language model correction levels, low / medium / high correction levels for each corpus, and a set of large language model correction annotation results (i.e., accurately annotated content after manual review) and the corresponding set of large language model pre-annotation results (i.e., initial annotation content of the dedicated model). Data associated with high correction levels in the large language model correction level set is then selected, extracting the manually correct annotations corresponding to these high-correction samples (i.e., the large language model correction annotation results) and the initial annotations of the dedicated model for that sample (i.e., the large language model pre-annotation results). These two sets are combined to form a difficult sample training set. Subsequently, this training set is used to incrementally train a set of dedicated models in the scalable model pool, such as research entity naming recognition models fine-tuned based on pre-trained architectures like BERT and RoBERTa. By focusing on samples where the dedicated model previously performed poorly (high-correction samples indicate large initial annotation bias), the model's capabilities in weak areas such as entity boundary judgment and easily confused type differentiation are specifically optimized, including research methods and theoretical principles, thereby continuously improving the accuracy of the dedicated model in extracting general research entities.

[0059] Key information is extracted from the associated feedback data, which includes a set of large language model correction levels, a set of large language model correction annotation results, and a corresponding set of large language model pre-annotation results. Data associated with high correction levels in the large language model correction level set is selected, i.e., the manually correct annotations and initial model pre-annotations corresponding to these high correction level samples are extracted and combined as pattern correction data. Subsequently, error pattern analysis is performed on the pattern correction data to identify common problems in the large language model when extracting general scientific research entities, such as research questions and algorithm models. For example, theoretical principles are misidentified as research methods, entity boundaries are not accurately defined, and specific types of entities are omitted. Based on these error patterns and the correct results of manual correction, the original preset prompt word template is optimized in a targeted manner. For example, clear distinctions between easily confused entity types are added, typical examples of entity boundary annotations are added, and the instruction requirement of not extracting vague or fictitious entities is strengthened. Finally, an optimized prompt word template is formed, which provides more accurate instruction guidance for subsequent calls to the large language model interface set to process difficult samples, thereby improving the accuracy of its scientific research entity extraction.

[0060] In one possible implementation, step S620 further includes:

[0061] Step S621: Perform error pattern recognition on the pattern correction data to obtain multiple error patterns and multiple associated pattern correction datasets.

[0062] Step S622: Based on the multiple association pattern correction dataset and the multiple error patterns, optimize the preset prompt word template to obtain an optimized prompt word template.

[0063] Specifically, error pattern recognition is performed on the extracted pattern correction data, namely the set of manually corrected annotations and pre-annotated results of the large language model corresponding to samples with high correction levels. By comparing the correctly manually annotated data with the initial extraction results of the large language model, i.e., the parts processed by the large model in the pre-annotated results, typical error types of the large model in general scientific research entity extraction are identified, forming multiple error patterns. For example, entity type confusion, such as misclassifying the algorithm model as a research method; entity boundary deviation, such as redundant or missing entity text fragments, such as misextracting the Transformer-based machine translation model as Transformer machine translation; entity omission, such as failing to identify entities such as the MATLAB software system and the experimental gas chromatograph equipment in the abstract. At the same time, the pattern correction data corresponding to each error pattern are classified to form multiple associated pattern correction datasets, such as the data subset corresponding to the entity type confusion error pattern and the data subset corresponding to the entity boundary deviation error pattern, providing data support for subsequent targeted optimization of prompt words.

[0064] Based on the identified error patterns, such as entity type confusion, entity boundary deviation, and entity omission, and the corresponding association pattern correction dataset, the preset prompt word template is adjusted in a targeted manner according to the characteristics of different error patterns and the logic of manual correction: For entity type confusion errors, such as misclassifying an algorithm model as a research method, the prompt word is supplemented with clear definitions and distinguishing examples of easily confused entity types, based on the correct annotation examples in the association pattern correction dataset, such as the specific process of solving a problem in the research method; the algorithm model is the core computing framework, such as the support vector machine algorithm; For entity boundary deviation errors, such as redundant or missing entity text fragments, standardized examples of entity boundary annotation are added based on the boundary correction records in the association dataset, such as correct annotation: software system MATLAB numerical calculation software; incorrect annotation: MATLAB numerical calculation, i.e., missing boundary, MATLAB numerical calculation software and tutorial, i.e., redundant boundary; For entity omission errors, the instructions for completely extracting seven types of general scientific research entities, i.e., research questions, research methods, etc., are strengthened by referring to the entity content completed in the association dataset. By integrating these targeted optimizations, an optimized prompt word template is ultimately formed, ensuring that when calling the large language model later, extraction errors can be reduced through clearer and more instructive commands, thereby improving the accuracy of general scientific research entity recognition.

[0065] In one possible implementation, step S700 further includes:

[0066] Step S710: Obtain the historical entity corpus construction data set, wherein the historical entity corpus construction data set includes multiple historical large model correction degree sets and multiple historical special model correction degree sets for multiple scientific research paper document types.

[0067] Step S720: Perform mean drift filtering on the multiple sets of historical large model correction degrees and the multiple sets of historical dedicated model correction degrees respectively to determine the concentrated values ​​of the multiple sets of historical large model correction degrees and the multiple sets of historical dedicated model correction degrees.

[0068] Step S730: When the multiple historical large model correction degree set values ​​are less than the corresponding historical dedicated model correction degree set values ​​among the multiple historical dedicated model correction degree set values, and the difference between the two meets the preset difference threshold, the corresponding scientific research paper document type is directly called through the large language model interface set to construct the corpus, as the model calling strategy optimization scheme, and the model calling strategy optimization scheme is embedded into the scalable model pool.

[0069] Specifically, a dataset is constructed by retrieving historical entity corpora from the database. This dataset consists of core data categorized and stored according to the type of research papers, such as computer science, biomedicine, materials engineering, and environmental science, from previous general research entity corpus construction tasks. Each research paper type corresponds to two sets of key feedback data: first, multiple historical large-scale model correction sets, recording the correction levels generated during manual review after extracting general research entities from the large-scale language models used in the past (e.g., GPT, DeepSeek), including high, medium, and low correction levels and quantified scores; second, multiple historical dedicated model correction sets, recording the correction levels generated during manual review after extracting dedicated models used in the past (e.g., BERT and RoBERTa-based fine-tuned research entity recognition models), belonging to the dedicated model set in the scalable model pool. Obtaining this categorized historical dataset provides data support for subsequent comparisons of different models' extraction performance across various types of documents and for optimizing model invocation strategies.

[0070] For the acquired historical data, it was grouped according to the type of scientific research papers, such as computer science and biomedicine. Mean drift filtering was then performed on both the historical large-scale model correction set and the historical specific model correction set within each group. This filtering method automatically eliminates outliers in the correction set by dynamically calculating the density center of the data distribution, such as extreme correction scores due to human error or abnormal correction records from a single special paper, ensuring that the final results accurately reflect the model's average performance on that type of literature. Subsequently, based on the filtered data, the central tendency value of the historical large-scale model correction for each type of literature was calculated. This central tendency value, such as the mean or median, represents the statistical central tendency of the degree of human correction of the large-scale model in the extracted results for that type of literature. This quantitatively reflects the quality of the large-scale model extraction and the central tendency value of the historical specific model correction, similarly reflecting the quality of the specific model extraction. This process not only avoids the interference of outlier data on model performance evaluation but also achieves a set of model performance indicators corresponding to each type of literature through categorized calculations. This provides accurate data support for subsequent comparisons of the advantages and disadvantages of different models on various types of literature and the development of targeted application strategies.

[0071] The model performance was compared one by one according to the type of scientific research paper. The concentration value of the historical large model correction degree was matched and compared with the concentration value of the historical dedicated model correction degree corresponding to the same type of paper. The lower the concentration value of the correction degree, the higher the quality of the model extraction results and the lower the manual correction cost. When the concentration value of the historical large model correction degree of a certain type of paper is less than the concentration value of the corresponding historical dedicated model correction degree, and the difference between the two reaches the preset difference threshold, such as a difference ≥ 0.3, it can be flexibly adjusted according to the corpus quality requirements and expected manual cost. This indicates that the extraction performance of the large language model for this type of paper is significantly better than that of the dedicated model. Using the dedicated model before conducting large model analysis will increase the workload of unnecessary manual correction.

[0072] At this point, the model invocation strategy for this type of scientific research paper is optimized to directly call the large model through the large language model interface set to construct the corpus, without the need for initial screening by a dedicated model, thus forming a model invocation strategy optimization scheme specific to this type of paper. Subsequently, this optimization scheme is embedded into the strategy configuration module of the scalable model pool. When this type of paper is imported later, the optimized invocation strategy will be automatically matched and executed, achieving precise scheduling of the optimal model for a type of paper.

[0073] Example 2, Figure 2 This is a schematic diagram of the structure of an electronic device provided in Embodiment 2 of this application, and a block diagram of an exemplary electronic device suitable for implementing the embodiments of the present invention. Figure 2 The electronic device shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention. Figure 2As shown, the electronic device includes a processor 21, a memory 22, an input device 23, and an output device 24; the number of processors 21 in the electronic device can be one or more. Figure 2 Taking a processor 21 as an example, the processor 21, memory 22, input device 23, and output device 24 in an electronic device can be connected via a bus or other means. Figure 2 Taking the example of a connection between China and Israel via a bus.

[0074] In embodiment three, the memory 22, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the method for constructing a general scientific research entity corpus based on scientific and technological literature in this application embodiment. The processor 21 executes various functional applications and data processing of the computer device by running the software programs, instructions, and modules stored in the memory 22, thereby realizing the above-mentioned method for constructing a general scientific research entity corpus based on scientific and technological literature.

[0075] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0076] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0077] This specification and accompanying drawings are merely illustrative examples of this application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application intends to include such modifications and variations.

Claims

1. A method for constructing a general scientific research entity corpus based on scientific and technological literature, characterized in that, The method includes: Build an extensible model pool, wherein the extensible model pool includes a dedicated model set and a large language model interface set; The set of scientific literature abstracts is automatically pre-labeled with general scientific research entities by calling a set of dedicated models, and the extracted set of general scientific research entities and confidence set are obtained. The general scientific research entities include at least one of research questions, research methods, algorithm models, software systems, metrics, instruments and equipment, theoretical principles and data. The confidence set is filtered and screened according to a preset confidence threshold, and the sample mapping is performed on the extracted general scientific research entity set and the scientific literature abstract set based on the filtering and screening results to obtain the difficult sample set and the general scientific research entity set extracted by the special model. The large language model interface set is called to extract the difficult sample set according to the preset prompt word template to obtain the extraction result set, and the extracted general scientific research entity set is fused based on the extraction result set to obtain the large language model pre-annotation result set. The corpus processing personnel review and correct the pre-annotated result set of the large language model, record the operations to obtain the operation record set and the pre-annotated result set of the large language model without operation, and identify the degree of correction based on the operation record set to obtain the corrected annotation result set and the correction degree set of the large language model. The corpus processing personnel reviewed and corrected the general scientific research entity set extracted by the special model, and extracted the identification and correction operation records to obtain the general scientific research entity set extracted by the special model without correction and the general scientific research entity set extracted by the special model with correction. The set of corrected annotation results of the large language model, the set of pre-annotated results of the unoperated large language model, the set of general scientific research entities extracted by the special model without correction, and the set of general scientific research entities extracted by the special model with correction are stored as general scientific research entity corpus. The set of correction degree of the large language model, the set of corrected annotation results of the large language model and the corresponding set of pre-annotated results of the large language model are associated and stored as feedback data. The feedback data is used to optimize the scalable model pool.

2. The method for constructing a general scientific research entity corpus based on scientific and technological literature as described in claim 1, characterized in that, The system uses a dedicated model set to automatically pre-annotate a set of scientific literature abstracts with general research entities, resulting in an extracted set of general research entities and a confidence set, including: The extracted general scientific research entity set is traversed for probability identification, and the identification results are averaged to obtain a confidence set. The probability identification refers to identifying the probability of all tokens constituting any general scientific research entity on its predicted label.

3. The method for constructing a general scientific research entity corpus based on scientific and technological literature as described in claim 1, characterized in that, The confidence set is filtered according to a preset confidence threshold, and based on the filtering results, the extracted general scientific entity set and the scientific literature abstract set are partitioned into sample mappings to obtain a set of difficult samples and a set of general scientific entity extracted by a dedicated model, including: Add confidence scores from the confidence score set that are less than the preset confidence threshold to the filter results; Based on the filtering results, the general scientific research entities extracted from the general scientific research entity set are mapped, and the mapping results are added to the difficult sample set. Add the scientific and technological literature abstracts in the collection of scientific and technological literature abstracts that have not been identified to the set of difficult samples; Add the set of general scientific research entities corresponding to confidence scores greater than or equal to the preset confidence threshold from the confidence score set to the set of general scientific research entities extracted by the dedicated model.

4. The method for constructing a general scientific research entity corpus based on scientific and technological literature as described in claim 1, characterized in that, Each operation record in the operation record set includes at least one of the following: modifying entity boundaries, modifying entity type, adding entity, and deleting entity.

5. The method for constructing a general scientific research entity corpus based on scientific and technological literature as described in claim 1, characterized in that, include: According to the preset operation weights, the operation record set is traversed to perform corrective scoring and identification, thereby obtaining a corrective scoring set; The correction score set is divided according to the preset correction degree division rules to obtain the large language model correction degree set; The preset correction level classification rule is that when the score is <0.5, it is considered a low correction level; When 0.5 ≤ score < 1.5, it is considered a medium correction level; A score of ≥1.5 indicates a high degree of correction.

6. The method for constructing a general scientific research entity corpus based on scientific and technological literature as described in claim 1, characterized in that, The set of large language model correction degrees, the set of large language model correction annotation results, and the corresponding set of large language model pre-annotation results are associated and stored as feedback data. The scalable model pool is then optimized using the feedback data, including: Extract the large language model correction annotation results and the corresponding large language model pre-annotation result set that are associated with high correction degree in the large language model correction degree set from the feedback data as training data, and perform incremental training on the dedicated model set; Extract the large language model correction annotation results and the corresponding large language model pre-annotation result set that are associated with high correction degree in the large language model correction degree set from the feedback data as pattern correction data. Optimize the preset prompt word template based on the pattern correction data to obtain the optimized prompt word template.

7. The method for constructing a general scientific research entity corpus based on scientific and technological literature as described in claim 6, characterized in that, Based on the pattern correction data, the preset prompt word template is optimized to obtain an optimized prompt word template, including: Error pattern identification is performed on the pattern correction data to obtain multiple error patterns and multiple associated pattern correction datasets; Based on the multiple association patterns, the dataset is corrected, and the multiple error patterns are used to optimize the preset prompt word template to obtain an optimized prompt word template.

8. The method for constructing a general scientific research entity corpus based on scientific and technological literature as described in claim 1, characterized in that, Also includes: Obtain a historical entity corpus to construct a dataset, wherein the historical entity corpus construction dataset includes multiple historical large model correction sets and multiple historical specific model correction sets for multiple scientific research paper document types; Mean drift filtering is performed on the multiple sets of historical large model correction degrees and the multiple sets of historical specific model correction degrees to determine the concentrated values ​​of the multiple sets of historical large model correction degrees and the multiple sets of historical specific model correction degrees. When the set value of the correction degree of the multiple historical large models is less than the set value of the correction degree of the multiple historical special models, and the difference between the two meets the preset difference threshold, the corresponding scientific research paper document type will be directly called through the large language model interface set to construct the corpus, as the model calling strategy optimization scheme, and the model calling strategy optimization scheme will be embedded into the scalable model pool.

9. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the method for constructing a general scientific research entity corpus based on scientific and technological literature as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method for constructing a general scientific research entity corpus based on scientific and technological literature as described in any one of claims 1-8.

Images