A scientific and technological literature retrieval method and device with clear semantic elements
By pre-constructing semantic feature tables and databases, parsing and predicting semantic elements in search statements, the problems of inaccurate literature search results and unclear matching reasons are solved, and more accurate and visual literature search results are achieved, improving user experience .
Patent Information
- Application Number
- CN202411087569.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-09
AI Technical Summary
The existing literature search methods have problems such as inaccurate results and unclear reasons for matching, which makes it difficult for users to quickly screen relevant documents.
By pre-constructing a semantic feature table and a literature feature database, analyzing the search statements entered by users, and predicting the semantic element attribution probability, including basic predictions, user predictions and historical predictions, matching and identifying the literature search results, ensuring that the attribution semantic elements of the search term appear The proportion is consistent with the predicted probability.
It improves the accuracy of literature search and the visibility of matching reasons, ensuring the relevance of search results and user satisfaction.
Smart Images

Figure CN118626633B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a scientific and technological document retrieval method and device with clear semantic elements. Background Art
[0002] With the growth of scientific and technological literature, how to efficiently and accurately obtain useful information from massive literature has become an important challenge for researchers. Traditional literature retrieval methods are usually based on keyword matching, which has some inherent limitations. In particular, when keywords are mentioned many times in the literature but are not the core content of the literature, the accuracy of the retrieval results is greatly affected. For example, in a research article, although the term "random forest" appears many times, it may only be mentioned as a comparative method or background knowledge, rather than the main research question or method of the article. In this case, the relevance of the retrieval results is low and cannot effectively meet the needs of users.
[0003] In addition, existing literature retrieval systems usually cannot clearly display the reasons for the match between the search results and the user's search statement (Query). It is difficult for users to understand why a certain document appears in the search results, which increases the difficulty of screening and analyzing documents. For example, when a user searches for a research question, the search results may include both documents directly related to the question and documents that only mention related terms in the method section, causing users to spend a lot of time to identify and screen relevant documents.
[0004] Therefore, the prior art has technical problems such as inaccurate document retrieval results and unclear reasons for matching retrieval results, making it difficult for users to quickly screen documents. Summary of the invention
[0005] The present application provides a scientific and technological literature retrieval method and device with clear semantic elements, which is used to solve the technical problems existing in the prior art of inaccurate literature retrieval results and unclear reasons for retrieval result matching.
[0006] In view of the above problems, the present application provides a scientific and technological literature retrieval method and device with clear semantic elements.
[0007] In a first aspect, the present application provides a method for searching scientific and technological documents with clear semantic elements, the method comprising:
[0008] Pre-constructing a semantic element table and a document element database, wherein the semantic element table includes multiple semantic elements, and the document element database includes element data of the multiple semantic elements in the document data;
[0009] Obtaining a search statement input by a user, and parsing it to obtain at least one search term;
[0010] Based on the semantic element table, predicting the semantic element attribution probability of the at least one search term to obtain at least one attribution semantic element set and at least one semantic element attribution probability set, wherein the semantic element attribution probability prediction includes basic prediction, user prediction, and historical prediction;
[0011] Based on the at least one search term, at least one set of belonging semantic elements and at least one set of semantic element belonging probabilities, search matching and identification are performed in the document element database to obtain a document search result, wherein the document search result includes the at least one search term, and the appearance ratio of at least one set of belonging semantic elements of the at least one search term is consistent with the at least one set of semantic element belonging probabilities.
[0012] In a second aspect, the present application provides a scientific and technological literature retrieval device for clarifying semantic elements, the device comprising:
[0013] A pre-construction unit, used for pre-constructing a semantic element table and a document element database, wherein the semantic element table includes multiple semantic elements, and the document element database includes element data of the multiple semantic elements in the document data;
[0014] A search parsing unit, used to obtain a search statement input by a user and parse it to obtain at least one search word;
[0015] An element attribution prediction unit, configured to perform semantic element attribution probability prediction on the at least one search term based on the semantic element table, and obtain at least one attribution semantic element set and at least one semantic element attribution probability set, wherein the semantic element attribution probability prediction includes basic prediction, user prediction, and historical prediction;
[0016] A retrieval unit is used to perform search matching and identification in the document element database according to the at least one search term, at least one set of belonging semantic elements and at least one set of semantic element belonging probabilities, so as to obtain a document retrieval result, wherein the document retrieval result includes the at least one search term, and the appearance ratio of at least one set of belonging semantic elements of the at least one search term is consistent with the at least one set of semantic element belonging probabilities.
[0017] One or more technical solutions provided in this application have at least the following technical effects or advantages:
[0018] The embodiment of the present application provides a method and device for scientific and technological literature retrieval with clear semantic elements. The method and device pre-construct a semantic element table including multiple semantic elements and a document element database including element data of the multiple semantic elements in the document data, parse the search statement input by the user to obtain the search term, and then predict the semantic element attribution probability of the search term from the basic prediction, user prediction, and historical prediction to obtain the attribution semantic element set and the semantic element attribution probability set, and then perform search matching and identification in the document element database to obtain the document search results, and the appearance ratio of multiple attribution semantic elements in the attribution semantic element set of the search term in the document search results is consistent with the semantic element attribution probability set. The present application constructs a semantic element classification standard, parses the search statement and performs semantic element prediction in three dimensions to improve the accuracy of semantic element prediction classification, thereby improving the accuracy of the final search results, and identifies the documents in the search results according to the predicted semantic elements, and displays the documents with the corresponding proportion of the search terms according to the probability of the predicted and classified multiple attribution semantic elements, so that the search results are clear and accurate, and the technical effect of improving the accuracy of the document search results and the visibility of the search matching reasons is achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 A schematic diagram of a process for a scientific and technological literature retrieval method with clear semantic elements is provided for this application;
[0020] Figure 2 A structural schematic diagram of a scientific and technological literature retrieval device with clear semantic elements is provided for this application.
[0021] Explanation of the reference numerals: pre-construction unit 11, retrieval and analysis unit 12, element attribution prediction unit 13, retrieval unit 14. DETAILED DESCRIPTION
[0022] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.
[0023] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.
[0024] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.
[0025] Embodiment 1: Figure 1 As shown, the present application provides a scientific and technological literature retrieval method with clear semantic elements, the method comprising:
[0026] S1: pre-constructing a semantic element table and a document element database, wherein the semantic element table includes multiple semantic elements, and the document element database includes element data of the multiple semantic elements in the document data;
[0027] In the embodiment of the present application, in order to perform scientific and technological literature retrieval based on clear semantic elements to improve the accuracy and efficiency of the literature retrieval, the semantic elements must first be defined to form a semantic element table.
[0028] In the embodiment of the present application, the semantic element table includes multiple semantic elements, which respectively define the semantics of different words and sentences in the document, such as research questions, method models, etc. Among them, a word and sentence may be classified as different semantic elements in different documents. For example, "random forest" can be classified as a research question or a method model. After clarifying the semantic elements, searching can improve the accuracy of the search and clarify the reasons for the search matching.
[0029] In the embodiment of the present application, the document element database includes specific element data of these semantic elements in the document data. For example, based on the above-mentioned multiple semantic elements, the words and sentences in all the documents in the database are classified and labeled as semantic elements, decomposed and mapped to the above-mentioned semantic elements, forming element data of multiple semantic elements, constructing index relationships, and forming a document element database.
[0030] Based on the constructed document element database, the search input statement and clear semantic elements can be used to retrieve and match documents with consistent search content and semantic elements, avoiding the retrieval of documents with inconsistent semantic elements. For example, if you need to search for documents that use "random forest" as a method model, but "random forest" in a certain document is not the method model of this article, but only mentioned or appears as a comparative content, and there is no semantic element marked as a method model, then this document will not be retrieved, improving the accuracy of the search.
[0031] Step S1 of the method provided in the embodiment of the present application includes:
[0032] Pre-constructing a variety of semantic elements, including research questions, method models, theoretical principles, measurement indicators, data materials, software systems and instruments and equipment;
[0033] According to the multiple semantic elements, semantic element data are extracted from the document data in the document library, and semantic element data vectors are constructed, which are indexed and stored with the corresponding documents to obtain a document element database.
[0034] In the embodiments of the present application, a variety of semantic elements are first pre-constructed. These elements include research questions, method models, theoretical principles, measurement indicators, data materials, software systems and instruments and equipment, which cover the vast majority of semantic elements of search terms in documents when searching for documents. Other semantic elements can also be set according to needs.
[0035] Then, semantic element data is extracted from the document data in the document database according to these semantic elements. For example, research questions, method models, theoretical principles, measurement indicators, data materials, software systems and instruments and equipment are extracted from each document in the document database, and semantic element data vectors are constructed and indexed and stored with the corresponding documents to finally obtain a document element database.
[0036] Among them, the research question is the core problem discussed in the literature or the scientific hypothesis proposed. For example, "the impact of crop yield" is a research question. The method model is the specific method or algorithm model used in the research, such as "random forest algorithm" or "Bayesian network". Theoretical principles are the basic theories and principles supporting the research, such as "Darwin's theory of evolution" or "the second law of thermodynamics". Metrics are indicators and standards used to evaluate research results, such as "precision", "recall rate" or "F1 score". Data materials are the original data and materials used in the research, such as "US Census data" or "gene expression data set". Software systems are software tools and platforms used for data processing and analysis, such as "MATLAB", "R language" or "TensorFlow". Instruments and equipment are experimental instruments and equipment used in the research, such as "nuclear magnetic resonance imaging machine" or "spectrometer".
[0037] Optionally, in the process of constructing a document element database, the documents in the document database may be preprocessed, such as text cleaning and word segmentation, and then entities in the documents, such as research questions, method models, theoretical principles, etc., are extracted, and the corresponding semantic elements are annotated, and further vectorized, such as using technologies such as Word2Vec, GloVe or BERT, to convert the extracted semantic elements into vector representations, and then index and store them with the corresponding documents. For example, a unique identifier can be established for each document and associated with its semantic element vector.
[0038] Among them, natural language processing technology can be used for text preprocessing, semantic analysis and semantic element labeling to ensure the accurate extraction of semantic elements and improve efficiency.
[0039] By pre-building a variety of semantic elements and extracting and vectorizing the document data, a document element database can be effectively constructed. This not only improves the efficiency and accuracy of document retrieval, but also enables users to locate the documents most relevant to their research needs more quickly.
[0040] S2: Obtaining a search statement input by a user, and parsing it to obtain at least one search term;
[0041] In the embodiment of the present application, the search sentence input by the user is obtained, for example, through the input interface of the document search engine, the search sentence input by the user is received and obtained, and the search sentence includes the content that the user wants to perform document search.
[0042] The search statement is parsed to obtain at least one search term. The parsing process includes processing, segmenting and keyword extraction of the search statement to extract key search information so as to accurately match relevant documents in the subsequent search process.
[0043] Step S2 in the method provided in the embodiment of the present application includes:
[0044] Get the search statement entered by the user;
[0045] Preprocessing the search statement to obtain at least one keyword;
[0046] The at least one keyword is expanded to obtain at least one search term, wherein the expansion process includes inputting the at least one keyword into an associated word library for associated indexing to determine whether there is an associated word, if so, adding the search term, if not, not adding it, and the associated word library includes multiple groups of associated phrases.
[0047] In the embodiment of the present application, a search statement input by a user is obtained and preprocessed to obtain at least one keyword.
[0048] Exemplarily, preprocessing includes text cleaning, word segmentation, etc. For example, if the search sentence is "how to use the random forest algorithm to predict the flight status of a drone", then text cleaning includes removing irrelevant characters, punctuation marks, and stop words (such as "how", "of", etc.) in the search sentence. For example, the processed search sentence becomes "use the random forest algorithm to predict the flight status of a drone".
[0049] Furthermore, the cleaned search sentences are decomposed into separate words or phrases. For example, using the word segmentation algorithm in natural language processing (NLP) technology, for example, "using random forest algorithm to predict the flight status of drones" is decomposed into "use", "random forest", "algorithm", "prediction", "drone", "flight", and "status".
[0050] Further, NER technology can be used to identify important entities in the search sentence. For example, "random forest" and "drone flight status", so that two keywords "random forest" and "drone flight status" are obtained. Other methods in the prior art can also be used to extract keywords from the search sentence to obtain at least one keyword.
[0051] In an embodiment of the present application, after processing and acquiring the keywords, the keywords input by the user may be represented by other similar words. For example, "convolutional neural network" can be represented by "CNN". If only the keyword "convolutional neural network" is used for retrieval, some documents marked with "CNN" may be missed. Therefore, at least one keyword is expanded.
[0052] In the embodiment of the present application, the expansion process includes inputting keywords into the associated word library for associated index judgment to determine whether there are associated words. The associated word library includes multiple groups of associated words, which can record multiple text expressions of the same actual content in the document search as a group of associated words, and then obtain multiple groups of associated words to construct an associated word library.
[0053] Input at least one keyword into the associated word library for associated indexing to determine whether there is an associated word. If so, add the associated word to the search word to obtain multiple search words. If not, do not add the associated word and use at least one keyword as the search word. In this way, obtain at least one search word.
[0054] The embodiment of the present application effectively parses the search sentence input by the user through the above steps, extracts keywords and performs expansion processing, and finally obtains rich and accurate search terms. These search terms will be used in the subsequent document search process to ensure that the search results are highly relevant to the user's needs.
[0055] S3: Based on the semantic element table, predict the semantic element attribution probability of the at least one search term to obtain at least one attribution semantic element set and at least one semantic element attribution probability set, wherein the semantic element attribution probability prediction includes basic prediction, user prediction, and historical prediction;
[0056] In the embodiment of the present application, based on the pre-constructed semantic element table, the semantic element attribution probability prediction is performed on the search terms obtained in the user's search sentence to obtain the semantic elements to which each search term may belong and the probability of attribution, and obtain at least one set of attribution semantic elements and at least one set of semantic element attribution probabilities. The semantic element attribution probability prediction includes three parts: basic prediction, user prediction and historical prediction.
[0057] Optionally, use a pre-trained BERT model to make preliminary classification predictions for search terms. The BERT model can understand the contextual semantics of search terms, thereby accurately predicting the probability of semantic element attribution of search terms. For each search term, the BERT model outputs a semantic element probability distribution. For example, the search term "random forest" may be predicted to belong to the method model 70% and the research question 30%.
[0058] In order to improve the accuracy of semantic element classification, predictions are made from three directions: basic prediction, user prediction, and historical prediction to improve the accuracy of semantic element classification.
[0059] Among them, basic prediction can use labeled sample data to classify the semantic elements of at least one search term, user prediction can predict the semantic element attribution of the current search term based on the user's historical search data, and historical prediction can predict the semantic element attribution of the search term based on all historical search data of all users. Semantic element classification prediction is performed through different sample data and different directions to improve the accuracy of semantic element classification.
[0060] The attribution probabilities of the semantic elements of the search terms obtained by basic prediction, user prediction and historical prediction can be further weighted to obtain the comprehensive probability that the search terms belong to different semantic elements, and obtain at least one set of attribution semantic elements and at least one set of semantic element attribution probabilities.
[0061] If at least one search term is multiple, each search term corresponds to an attribution semantic element set and a semantic element attribution probability set, forming an attribution semantic element combination of the search term combination.
[0062] Based on the semantic element table, the semantic element attribution probability of the search terms in the user's search statement is accurately predicted to obtain the set of attribution semantic elements and the set of semantic element attribution probability. This information will be used for subsequent literature retrieval and matching to improve the relevance and accuracy of the search results.
[0063] Step S3 of the method provided in the embodiment of the present application includes:
[0064] Collecting a sample search term set, and annotating each sample search term with a semantic element according to the semantic element table to obtain a sample attribution semantic element set;
[0065] Using the sample search term set and the sample attribution semantic element set, constructing a basic semantic element classification prediction model, performing semantic element attribution probability prediction on the at least one search term, obtaining at least one basic attribution semantic element set and at least one basic semantic element attribution probability set, wherein the basic semantic element classification prediction model includes a plurality of basic semantic element classification prediction branches;
[0066] Performing user prediction and history prediction to obtain at least one user attribution semantic element set and at least one user semantic element attribution probability set, and at least one history attribution semantic element set and at least one history semantic element attribution probability set;
[0067] Integrate the at least one basic attribution semantic element set, at least one user attribution semantic element set and at least one historical attribution semantic element set to obtain at least one attribution semantic element set, perform weighted calculation on the basic semantic element attribution probability, user semantic element attribution probability and historical semantic element attribution probability of at least one search term belonging to each attribution semantic element to obtain at least one semantic element attribution probability set.
[0068] In the embodiment of the present application, a sample search term set is collected, and each sample search term is annotated with semantic elements according to a predefined semantic element table to construct a basic semantic element classification prediction model. Then, the semantic element attribution probability of the user search term is predicted, and the user prediction and historical prediction are combined, and finally the various prediction results are integrated for weighted calculation to obtain an accurate semantic element attribution probability.
[0069] Specifically, a large number of search terms are collected from the literature data, and the context of the search terms can also be included to form a sample search term set. These search terms represent the search intentions of users in various contexts, such as "random forest", "crops", etc.
[0070] Furthermore, semantic elements are annotated for each sample search term based on the various semantic elements in the semantic element table. Specifically, the annotation can be based on the semantic elements of the sample search terms in the literature where the sample search terms are collected. For example, the search term "random forest" is annotated as a method model, and the search term "drone flight status" is annotated as a research question. In this way, a set of sample attribution semantic elements is obtained.
[0071] Among them, a sample search term may belong to different semantic elements in different documents. For example, "random forest" may be a method model or a research question. However, with a large amount of sample data, the semantic element to which a sample search term belongs can form a certain probability distribution. For example, "random forest" is most likely a method model.
[0072] A sample search term set and a sample attribution semantic element set are used to construct a basic semantic element classification prediction model, and a semantic element attribution probability prediction is performed on at least one search term to perform basic prediction to obtain at least one basic attribution semantic element set and at least one basic semantic element attribution probability set.
[0073] Among them, the basic semantic element classification prediction model includes multiple basic semantic element classification prediction branches, which respectively predict the semantic element attribution probability of a search term, and can obtain multiple semantic element classification results of multiple basic semantic element classification prediction branches as basic attribution semantic elements to form a basic attribution semantic element set of the search term. In addition, according to the proportion of different basic attribution semantic elements in the set, the basic semantic element attribution probability can be assigned to each different basic attribution semantic element to obtain a corresponding basic semantic element attribution probability set.
[0074] The step of "using the sample search term set and the sample attribution semantic element set to construct a basic semantic element classification prediction model, and predicting the semantic element attribution probability of the at least one search term" in the method provided in the embodiment of the present application includes:
[0075] Randomly extract a preset proportion of data from the sample search term set and the sample attribution semantic element set, and construct a first basic semantic element classification prediction branch based on the BERT model;
[0076] Continue to randomly extract a preset proportion of data from the sample search term set and the sample attribution semantic element set multiple times, build other multiple basic semantic element classification prediction branches based on the BERT model, integrate the multiple basic semantic element classification prediction branches, and obtain a basic semantic element classification prediction model;
[0077] The at least one search term is input into the basic semantic element classification prediction model respectively, and at least one basic attribution semantic element set is obtained through classification prediction. At least one basic semantic element attribution probability set is calculated based on the appearance ratio of each basic attribution semantic element.
[0078] In the embodiment of the present application, by randomly extracting data from the sample search term set and the sample attribution semantic element set, multiple basic semantic element classification prediction branches are constructed based on the BERT model, and these branches are integrated to form a basic semantic element classification prediction model. Then, the search term is input into the model, and multiple classification predictions of multiple branches are performed to obtain the basic attribution semantic element set and the basic semantic element attribution probability set.
[0079] In order to classify and predict as many semantic elements as possible to which the search term may belong, a plurality of different basic semantic element classification prediction branches are constructed in the embodiment of the present application to perform basic semantic element classification prediction. In order to ensure that the plurality of basic semantic element classification prediction branches are different, a certain proportion of training data is randomly selected from the sample search term set and the sample attribution semantic element set to construct different basic semantic element classification prediction branches.
[0080] Randomly extract a preset proportion of data from the sample search term set and the sample attribute semantic element set. For example, randomly extract a preset proportion of 40%-70% (i.e., 400-700) from 1,000 samples as training data. The extracted training data includes multiple sets of corresponding sample search terms and sample attribute semantic elements. By randomly extracting data sets multiple times, it is ensured that the model has sufficient diversity and generalization ability.
[0081] For example, the training is performed based on the BERT model in the prior art, which can understand the semantic features of the search terms. Alternatively, other neural networks can also be used for training.
[0082] Exemplarily, based on the selected BERT model, a randomly selected sample data set is used to train the BERT model, and the first basic semantic element classification prediction branch is trained. For example, the pre-trained BERT model is loaded using the Transformers library of Hugging Face, and word embedding is performed, and the vector after word embedding is input into the BERT model. The input search word vector is encoded through a multi-layer Transformer structure to capture the deep semantic features of the search word. The output layer of the model is the semantic element category. A deep learning framework such as TensorFlow or PyTorch is used for model training, and the parameters of the BERT model are optimized. The cross entropy loss function is used to measure the difference between the predicted probability distribution and the actual label. The Adam optimizer is used to minimize the loss function, and the model parameters are adjusted to improve the classification accuracy until the basic semantic element classification prediction branch converges, for example, the accuracy meets the requirements, for example, reaches 90%. The accuracy of the basic semantic element classification prediction branch can be further verified. After the verification is qualified, the training is completed, otherwise the training continues.
[0083] Based on the same steps, continue to randomly extract a preset proportion of data from the sample search term set and the sample attribution semantic element set multiple times, build other multiple basic semantic element classification prediction branches based on the BERT model, integrate, for example, connect the input layer of multiple basic semantic element classification prediction branches, and obtain a basic semantic element classification prediction model.
[0084] In an embodiment of the present application, at least one search term obtained by analyzing the aforementioned content is respectively input into the basic semantic element classification prediction model, and each search term is input into multiple basic semantic element classification prediction branches. Since the training data of each basic semantic element classification prediction branch is different, multiple different semantic elements can be classified and obtained respectively, and at least one basic attribution semantic element set can be obtained.
[0085] According to the occurrence ratio of each basic semantic element in each basic semantic element set, at least one basic semantic element attribution probability set is calculated. For example, among 10 branches, 7 branches predict that "random forest" is attributed to the method model, and the occurrence ratio of the method model is 70%, and the corresponding basic semantic element attribution probability is 0.7.
[0086] Through the above steps, the search terms entered by the user are respectively input into the basic semantic element classification prediction model for classification prediction, and the basic semantic element attribution probability set is calculated based on the occurrence ratio of each basic attribution semantic element. This process not only improves the accuracy of the prediction results, but also ensures the efficient matching of the user's search terms and the document semantic elements.
[0087] Furthermore, after completing the basic prediction, user prediction and historical prediction are performed to obtain at least one user attribution semantic element set and at least one user semantic element attribution probability set, as well as at least one historical attribution semantic element set and at least one historical semantic element attribution probability set.
[0088] The step of "performing user prediction and historical prediction" in the method provided in the embodiment of the present application includes:
[0089] Acquire a user search sentence set of the user, parse to obtain a user search word set, and annotate to obtain a user attribute semantic element set according to the selection result of the user search;
[0090] Using the user search term set and the user attribute semantic element set, constructing a user semantic element classification prediction model, wherein the user semantic element classification prediction model includes a plurality of user semantic element classification prediction branches;
[0091] Using the user semantic element classification prediction model, predicting the semantic element attribution probability of the at least one search term, and obtaining at least one user attribution semantic element set and at least one user semantic element attribution probability set;
[0092] Obtain a set of historical search sentences within a historical period, parse to obtain a set of historical search terms, and annotate to obtain a set of historical attribution semantic elements;
[0093] Using the historical search term set and the historical attribution semantic element set, construct a historical semantic element classification prediction model, wherein the historical semantic element classification prediction model includes multiple historical semantic element classification prediction branches;
[0094] The historical semantic element classification prediction model is used to predict the semantic element attribution probability of the at least one search term, and at least one historical attribution semantic element set and at least one historical semantic element attribution probability set are obtained.
[0095] In the embodiment of the present application, user prediction and historical prediction are performed based on the search data within the user's historical time and the search data within the history of all users as sample data to obtain the semantic elements of at least one search term.
[0096] In the embodiment of the present application, all search statements entered by the user within a specific time period in history, such as search statements for literature search within one year, are collected to form a user search statement set. The user search statements are cleaned, segmented, and keyword extracted based on the methods in the above content to form a user search word set.
[0097] According to the selection results of the user's search, the semantic elements of the user's search terms are marked to form a set of user-attributed semantic elements. Exemplarily, after the user retrieves multiple documents, the semantic elements of the user's search terms in the documents selected by the user are marked to obtain a set of user-attributed semantic elements. For example, if the user selects a document with "random forest" as the research question in the search, the semantics of "random forest" will be seriously marked as the research question. The user's document selection can be determined based on the document that the user has read the longest, or the document with the highest search accuracy input by the user can be obtained through the evaluation system.
[0098] Based on the same construction process of the basic semantic element classification prediction model in the aforementioned content, data is randomly extracted from the user search term set and the user attribute semantic element set, and multiple samplings are performed. Based on the BERT model, the data sampled each time is trained to construct multiple user semantic element classification prediction branches to form a comprehensive user semantic element classification prediction model.
[0099] At least one search term is input into the user semantic element classification prediction model, and semantic element attribution probability prediction is performed to obtain at least one user attribution semantic element set and at least one user semantic element attribution probability set. Since the user semantic element classification prediction model and the basic semantic element classification prediction model have different training data, the prediction results are also different.
[0100] Furthermore, the search statements of all users in the document retrieval system in the historical time are collected to form a historical search statement set. The historical search statements are preprocessed and parsed such as word segmentation to extract keywords and form a historical search word set.
[0101] According to the semantic elements of the historical search terms in the selection results of the searching user in the historical search, a set of historical attribution semantic elements is annotated. For example, when a user obtains multiple documents when searching for "random forest", and the attribution semantic element of "random forest" in the final selected document is method model, it will be annotated as method model.
[0102] A historical semantic element classification prediction model is constructed using a historical search term set and a historical attribution semantic element set. Based on the same steps in the previous content, data is randomly extracted from the historical search term set and the historical attribution semantic element set, and multiple samplings are performed. Based on the BERT model, the data sampled each time is trained to construct multiple historical semantic element classification prediction branches to form a comprehensive historical semantic element classification prediction model.
[0103] Input the current at least one search term into the historical semantic element classification prediction model, perform semantic element attribution probability prediction, obtain a historical attribution semantic element set and a historical semantic element attribution probability set, and complete the historical prediction.
[0104] After completing basic prediction, user prediction, and historical prediction, at least one basic attribution semantic element set and at least one basic semantic element attribution probability set, at least one user attribution semantic element set and at least one user semantic element attribution probability set, and at least one historical attribution semantic element set and at least one historical semantic element attribution probability set are obtained.
[0105] All basic attribution semantic elements, user attribution semantic elements and historical attribution semantic elements of at least one search term in the semantic element classification prediction result are extracted to obtain at least one attribution semantic element set.
[0106] Furthermore, the attribution probability of each attribution semantic element in the basic semantic element attribution probability set, user semantic element attribution probability set and historical semantic element attribution probability set of the corresponding search term is obtained, and weighted calculation is performed to obtain a comprehensive semantic element attribution probability, and then a semantic element attribution probability set of a search term is obtained. Finally, at least one semantic element attribution probability set of at least one search term is obtained. For example, for the search term "random forest", the basic semantic element classification prediction model predicts that its attribution probability is 70% for the method model, 20% for the research problem, and 10% for the theoretical principle; the user semantic element classification prediction model predicts that its attribution probability is 60% for the method model, 30% for the research problem, and 10% for the theoretical principle; the historical semantic element classification prediction model predicts that its attribution probability is 75% for the method model, 15% for the research problem, and 10% for the theoretical principle. Then for "random forest", the final attribution probability of its semantic element being the method model can be calculated as: .
[0107] The weighted probability can be set by yourself, for example, the weights of the basic semantic element attribution probability, the user semantic element attribution probability and the historical semantic element attribution probability are 0.3, 0.4 and 0.3 respectively. If the sum of the weighted semantic element attribution probabilities of multiple attributable semantic elements of a search term after weighting is not 1, the semantic element attribution probabilities are redistributed according to the size of the weighted semantic element attribution probabilities so that their sum is 1.
[0108] Through the above steps, the basic attribute semantic element set, user attribute semantic element set and historical attribute semantic element set are integrated, and the search terms are weighted to obtain the final semantic element attribute probability set. This process ensures the comprehensiveness and accuracy of the prediction results, and improves the relevance of literature retrieval and user satisfaction.
[0109] S4: Based on the at least one search term, at least one set of belonging semantic elements and at least one set of semantic element belonging probabilities, search matching and identification are performed in the document element database to obtain a document search result, wherein the document search result includes the at least one search term, and the appearance ratio of at least one set of belonging semantic elements of the at least one search term is consistent with the at least one set of semantic element belonging probabilities.
[0110] In an embodiment of the present application, based on at least one search term, at least one set of attribution semantic elements, and at least one set of semantic element attribution probabilities obtained through processing, a search is performed in a document element database to obtain documents that include these search terms and whose semantic elements match at least one set of attribution semantic elements.
[0111] In order to make the speech matching of the document search clear, the documents can be marked based on the search terms included in the retrieved documents and the corresponding attributive semantic elements, thereby improving the visibility of the search results.
[0112] In order to make the search results accurate and reasonable, the occurrence ratio of multiple semantic elements in at least one semantic element set of at least one search term in the document search results is consistent with the multiple semantic element attribution probabilities in at least one semantic element attribution probability set. In this way, the accuracy and rationality of the search results are improved.
[0113] Step S4 in the method provided in the embodiment of the present application includes:
[0114] According to the at least one search term and the at least one set of attributive semantic elements, searching and matching are performed in the document element database to obtain a plurality of matching documents including the at least one search term and whose semantic elements match the at least one set of attributive semantic elements;
[0115] According to the at least one semantic element attribution probability set, the plurality of matching documents are screened to obtain a plurality of search documents, wherein the occurrence ratio of the plurality of attribution semantic elements of each search term in the plurality of search documents is consistent with the plurality of semantic element attribution probabilities;
[0116] According to the search terms and corresponding semantic elements appearing in each search document, each search document is marked to obtain the document search results.
[0117] In the embodiment of the present application, according to at least one search term and at least one set of attribution semantic elements, a search and matching is performed in the document element database, and documents including any search term and the semantic elements of the search term and any attribution semantic elements in at least one set of attribution semantic elements are screened out to obtain multiple matching documents. For example, the set of attribution semantic elements corresponding to the search term "random forest" includes method models and research questions, then the documents including "random forest" and the semantic elements of which are method models or research questions are indexed in the document element database.
[0118] According to at least one semantic element attribution probability set, multiple matching documents are screened so that the proportion of multiple semantic elements of the search terms in the multiple documents is the same as the multiple semantic element attribution probabilities in the semantic element attribution probability set of the search terms. For example, the attribution probability of the search term "random forest" in the method model is 70%, and the attribution probability in the research question is 30%. Then, the proportion of the semantic elements of random forest in the method model and the research question in the multiple documents is 70% and 30% respectively, and multiple search documents are obtained.
[0119] Among them, if there is one search term, it will be screened according to one search term. If there are multiple search terms, each search term can be screened separately, or multiple search terms can be screened in combination, as long as the appearance ratio of multiple semantic elements of each search term is consistent with the probability of attribution of multiple semantic elements. If the combined screening cannot meet the requirements, it can be converted to a single screening.
[0120] According to the search terms and corresponding semantic elements appearing in each search document, each search document is marked and displayed in the search results, so that the search matching reasons are clear and the document search results are obtained.
[0121] Optionally, other document search and ranking methods in the prior art may be used to rank the multiple retrieved documents. For example, the search documents including the semantic elements with a larger semantic element attribution probability of the search terms are ranked higher, and the search documents corresponding to the semantic elements with a smaller semantic element attribution probability are ranked lower. If multiple search documents contain the same search terms and the same semantic elements, they are ranked randomly. Optionally, the search documents with more search terms and a larger sum of semantic element attribution probabilities of the semantic elements of the search terms may be ranked higher.
[0122] Through the above steps, it is possible to perform search matching and identification in the document element database, generate document search results containing the search terms, and ensure that the appearance ratio of multiple semantic elements of the search terms in the document search results is consistent with the attribution probability of multiple semantic elements in the semantic element attribution probability set. This process not only improves the relevance of the search results, but also enhances the user's understanding and application of the document, making the reasons for the search matching clear and more suitable for the user's search needs.
[0123] The present application provides a method and device for searching scientific and technological documents with clear semantic elements, which has the following technical effects:
[0124] 1. A semantic element table was constructed, and a document element database was established to lay the foundation for document retrieval based on clear semantic elements.
[0125] 2. Parse the search sentences and predict the probabilities of belonging to different semantic elements. In addition, introduce the historical search data of users and all users to predict semantic elements in three dimensions. In addition, introduce related words and combine ensemble learning to determine the probability of semantic element attribution based on the proportion of semantic elements predicted by different branch models, thereby improving the accuracy and comprehensiveness of semantic element prediction and the accuracy of semantic element prediction classification, thereby improving the accuracy of the final search results.
[0126] 3. Based on the established semantic element table and document element database, the documents in the search results are marked according to the semantic elements of the search terms in the searched documents, so that the search results are clear, and the corresponding proportion of documents are displayed according to the probability of multiple semantic elements of the search terms that are finally predicted, so that the search results are more accurate and meet user needs.
[0127] Embodiment 2: Based on the same inventive concept as the method for searching scientific and technological documents with clear semantic elements in the above embodiment, Figure 2 As shown, the present application provides a scientific and technological literature retrieval device with clear semantic elements, the device comprising:
[0128] A pre-construction unit 11 is used to pre-construct a semantic element table and a document element database, wherein the semantic element table includes multiple semantic elements, and the document element database includes element data of the multiple semantic elements in the document data;
[0129] A search parsing unit 12, used to obtain a search statement input by a user and parse it to obtain at least one search word;
[0130] An element attribution prediction unit 13 is used to perform semantic element attribution probability prediction on the at least one search term based on the semantic element table to obtain at least one attribution semantic element set and at least one semantic element attribution probability set, wherein the semantic element attribution probability prediction includes basic prediction, user prediction, and historical prediction;
[0131] The retrieval unit 14 is used to perform search matching and identification in the document element database according to the at least one search term, at least one set of belonging semantic elements and at least one set of semantic element belonging probabilities, so as to obtain a document retrieval result, wherein the document retrieval result includes the at least one search term, and the appearance ratio of at least one set of belonging semantic elements of the at least one search term is consistent with the at least one set of semantic element belonging probabilities.
[0132] Furthermore, the pre-construction unit 11 further comprises the following steps: pre-constructing a plurality of semantic elements, wherein the plurality of semantic elements comprises research questions, method models, theoretical principles, measurement indicators, data materials, software systems and instruments and equipment;
[0133] According to the multiple semantic elements, semantic element data are extracted from the document data in the document library, and semantic element data vectors are constructed, which are indexed and stored with the corresponding documents to obtain a document element database.
[0134] Furthermore, the search and analysis unit 12 further includes the following steps: obtaining a search statement input by a user;
[0135] Preprocessing the search statement to obtain at least one keyword;
[0136] The at least one keyword is expanded to obtain at least one search term, wherein the expansion process includes inputting the at least one keyword into an associated word library for associated indexing to determine whether there is an associated word, if so, adding the search term, if not, not adding it, and the associated word library includes multiple groups of associated phrases.
[0137] Furthermore, the element attribution prediction unit 13 further includes the following steps: collecting a set of sample search terms, and annotating each sample search term with a semantic element according to the semantic element table to obtain a set of sample attribution semantic elements;
[0138] Using the sample search term set and the sample attribution semantic element set, constructing a basic semantic element classification prediction model, performing semantic element attribution probability prediction on the at least one search term, obtaining at least one basic attribution semantic element set and at least one basic semantic element attribution probability set, wherein the basic semantic element classification prediction model includes a plurality of basic semantic element classification prediction branches;
[0139] Performing user prediction and history prediction to obtain at least one user attribution semantic element set and at least one user semantic element attribution probability set, and at least one history attribution semantic element set and at least one history semantic element attribution probability set;
[0140] Integrate the at least one basic attribution semantic element set, at least one user attribution semantic element set and at least one historical attribution semantic element set to obtain at least one attribution semantic element set, perform weighted calculation on the basic semantic element attribution probability, user semantic element attribution probability and historical semantic element attribution probability of at least one search term belonging to each attribution semantic element to obtain at least one semantic element attribution probability set.
[0141] The sample search term set and the sample attribution semantic element set are used to construct a basic semantic element classification prediction model, and the semantic element attribution probability prediction is performed on the at least one search term, including:
[0142] Randomly extract a preset proportion of data from the sample search term set and the sample attribution semantic element set, and construct a first basic semantic element classification prediction branch based on the BERT model;
[0143] Continue to randomly extract a preset proportion of data from the sample search term set and the sample attribution semantic element set multiple times, build other multiple basic semantic element classification prediction branches based on the BERT model, integrate the multiple basic semantic element classification prediction branches, and obtain a basic semantic element classification prediction model;
[0144] The at least one search term is input into the basic semantic element classification prediction model respectively, and at least one basic attribution semantic element set is obtained through classification prediction. At least one basic semantic element attribution probability set is calculated based on the appearance ratio of each basic attribution semantic element.
[0145] wherein, a user search sentence set of the user is obtained, a user search word set is obtained by parsing, and a user attribute semantic element set is obtained by annotating according to the selection result of the user search;
[0146] Using the user search term set and the user attribute semantic element set, constructing a user semantic element classification prediction model, wherein the user semantic element classification prediction model includes a plurality of user semantic element classification prediction branches;
[0147] Using the user semantic element classification prediction model, predicting the semantic element attribution probability of the at least one search term, and obtaining at least one user attribution semantic element set and at least one user semantic element attribution probability set;
[0148] Obtain a set of historical search sentences within a historical period, parse to obtain a set of historical search terms, and annotate to obtain a set of historical attribution semantic elements;
[0149] Using the historical search term set and the historical attribution semantic element set, construct a historical semantic element classification prediction model, wherein the historical semantic element classification prediction model includes multiple historical semantic element classification prediction branches;
[0150] The historical semantic element classification prediction model is used to predict the semantic element attribution probability of the at least one search term, and at least one historical attribution semantic element set and at least one historical semantic element attribution probability set are obtained.
[0151] Furthermore, the retrieval unit 14 further comprises the following steps: performing retrieval matching in the document element database according to the at least one search term and the at least one belonging semantic element set, and obtaining a plurality of matching documents including the at least one search term and the semantic elements matching the at least one belonging semantic element set;
[0152] According to the at least one semantic element attribution probability set, the plurality of matching documents are screened to obtain a plurality of search documents, wherein the occurrence ratio of the plurality of attribution semantic elements of each search term in the plurality of search documents is consistent with the plurality of semantic element attribution probabilities;
[0153] According to the search terms and corresponding semantic elements appearing in each search document, each search document is marked to obtain the document search results.
[0154] Through the above-mentioned detailed description of a method for retrieving scientific and technological literature with clear semantic elements in this specification, those skilled in the art can clearly know a method and device for retrieving scientific and technological literature with clear semantic elements in this embodiment. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.
[0155] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A scientific literature retrieval method with clear semantic elements, It is characterized in that The method comprises: Pre-constructing a semantic element table and a document element database, wherein the semantic element table includes multiple semantic elements, and the document element database includes element data of the multiple semantic elements in the document data; Obtaining a search statement input by a user, and parsing it to obtain at least one search term; Based on the semantic element table, predicting the semantic element attribution probability of the at least one search term to obtain at least one attribution semantic element set and at least one semantic element attribution probability set, wherein the semantic element attribution probability prediction includes basic prediction, user prediction, and historical prediction; According to the at least one search term, the at least one semantic element set and the plurality of semantic element attribution probability sets, search matching and identification are performed in the document element database to obtain a document search result, wherein the document search result includes the at least one search term, and the appearance ratio of at least one semantic element set of the at least one search term is consistent with the at least one semantic element attribution probability set; Based on the semantic element table, predicting the semantic element attribution probability of the at least one search term to obtain at least one attribution semantic element set and at least one semantic element attribution probability set includes: Collecting a sample search term set, and annotating each sample search term with a semantic element according to the semantic element table to obtain a sample attribution semantic element set; Using the sample search term set and the sample attribution semantic element set, constructing a basic semantic element classification prediction model, performing semantic element attribution probability prediction on the at least one search term, obtaining at least one basic attribution semantic element set and at least one basic semantic element attribution probability set, wherein the basic semantic element classification prediction model includes a plurality of basic semantic element classification prediction branches; Performing user prediction and history prediction to obtain at least one user attribution semantic element set and at least one user semantic element attribution probability set, and at least one history attribution semantic element set and at least one history semantic element attribution probability set; Integrate the at least one basic attribution semantic element set, the at least one user attribution semantic element set and the at least one historical attribution semantic element set to obtain at least one attribution semantic element set, and perform weighted calculation on the basic semantic element attribution probability, the user semantic element attribution probability and the historical semantic element attribution probability of the at least one search term belonging to each attribution semantic element to obtain at least one semantic element attribution probability set; Perform user forecasting and historical forecasting, including: Acquire a user search sentence set of the user, parse to obtain a user search word set, and annotate to obtain a user attribute semantic element set according to the selection result of the user search; Using the user search term set and the user attribute semantic element set, constructing a user semantic element classification prediction model, wherein the user semantic element classification prediction model includes a plurality of user semantic element classification prediction branches; Using the user semantic element classification prediction model, predicting the semantic element attribution probability of the at least one search term, and obtaining at least one user attribution semantic element set and at least one user semantic element attribution probability set; Obtain a set of historical search sentences within a historical period, parse to obtain a set of historical search terms, and annotate to obtain a set of historical attribution semantic elements; Using the historical search term set and the historical attribution semantic element set, construct a historical semantic element classification prediction model, wherein the historical semantic element classification prediction model includes multiple historical semantic element classification prediction branches; The historical semantic element classification prediction model is used to predict the semantic element attribution probability of the at least one search term, and at least one historical attribution semantic element set and at least one historical semantic element attribution probability set are obtained.
2. The scientific and technological literature retrieval method for clarifying semantic elements according to claim 1, It is characterized in that Pre-built semantic element tables and document element databases, including: Pre-constructing a variety of semantic elements, including research questions, method models, theoretical principles, measurement indicators, data materials, software systems and instruments and equipment; According to the multiple semantic elements, semantic element data are extracted from the document data in the document library, and semantic element data vectors are constructed, which are indexed and stored with the corresponding documents to obtain a document element database.
3. The scientific and technological literature retrieval method for clarifying semantic elements according to claim 1, It is characterized in that Obtain the search statement input by the user and parse it to obtain at least one search term, including: Get the search statement entered by the user; Preprocessing the search statement to obtain at least one keyword; The at least one keyword is expanded to obtain at least one search term, wherein the expansion process includes inputting the at least one keyword into an associated word library for associated indexing to determine whether there is an associated word, if so, adding the search term, if not, not adding it, and the associated word library includes multiple groups of associated phrases.
4. The scientific and technological literature retrieval method for clarifying semantic elements according to claim 1, It is characterized in that The sample search term set and the sample attribution semantic element set are used to construct a basic semantic element classification prediction model, and the semantic element attribution probability prediction is performed on the at least one search term, including: Randomly extract a preset proportion of data from the sample search term set and the sample attribution semantic element set, and construct a first basic semantic element classification prediction branch based on the BERT model; Continue to randomly extract a preset proportion of data from the sample search term set and the sample attribution semantic element set multiple times, build other multiple basic semantic element classification prediction branches based on the BERT model, integrate the multiple basic semantic element classification prediction branches, and obtain a basic semantic element classification prediction model; The at least one search term is input into the basic semantic element classification prediction model respectively, and at least one basic attribution semantic element set is obtained through classification prediction. At least one basic semantic element attribution probability set is calculated based on the appearance ratio of each basic attribution semantic element.
5. The scientific and technological literature retrieval method for clarifying semantic elements according to claim 1, It is characterized in that According to the at least one search term, the at least one semantic element set and the plurality of semantic element attribution probability sets, searching, matching and marking are performed in the document element database to obtain a document search result, including: According to the at least one search term and the at least one set of attributive semantic elements, searching and matching are performed in the document element database to obtain a plurality of matching documents including the at least one search term and whose semantic elements match the at least one set of attributive semantic elements; According to the plurality of semantic element attribution probability sets, the plurality of matching documents are screened to obtain a plurality of search documents, wherein the occurrence ratio of the plurality of attribution semantic elements of each search term in the plurality of search documents is consistent with the plurality of semantic element attribution probabilities; According to the search terms and corresponding semantic elements appearing in each search document, each search document is marked to obtain the document search results.
6. A scientific and technological literature retrieval device with clear semantic elements, It is characterized in that The steps for implementing the method for searching scientific and technological documents with clear semantic elements as described in any one of claims 1 to 5, wherein the device for searching scientific and technological documents with clear semantic elements comprises: A pre-construction unit, used for pre-constructing a semantic element table and a document element database, wherein the semantic element table includes multiple semantic elements, and the document element database includes element data of the multiple semantic elements in the document data; A search parsing unit, used to obtain a search statement input by a user and parse it to obtain at least one search word; An element attribution prediction unit, configured to perform semantic element attribution probability prediction on the at least one search term based on the semantic element table, and obtain at least one attribution semantic element set and at least one semantic element attribution probability set, wherein the semantic element attribution probability prediction includes basic prediction, user prediction, and historical prediction; A retrieval unit is used to perform search matching and identification in the document element database according to the at least one search term, at least one set of belonging semantic elements and multiple sets of semantic element belonging probabilities, so as to obtain a document retrieval result, wherein the document retrieval result includes the at least one search term, and the appearance ratio of at least one set of belonging semantic elements of the at least one search term is consistent with the at least one set of semantic element belonging probability.
