Method and system for predicting scholars' homepages and languages ​​used

Abstract

This application proposes a method for predicting a scholar's homepage and the language used, relating to the field of information processing technology. The method comprises obtaining multiple webpages related to the scholar and extracting features of each webpage to generate a webpage feature vector matrix corresponding to the multiple webpages; learning the semantic feature vectors of the text in each webpage through a preset first training model to generate a semantic feature vector matrix corresponding to the multiple webpages; predicting the scholar's homepage by combining the webpage feature vector matrix and the semantic feature vector matrix; designing a question in the language used by the scholar, and obtaining an answer to the question through a preset second training model based on the text of the predicted scholar's homepage. The method utilizes the retrieval capabilities of an Internet search engine to train the text semantic information of the retrieved webpages, thereby improving the prediction effect of the scholar's homepage and language.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a method and system for predicting a scholar's homepage and the language used. Background Art

[0002] Currently, the evaluation and discovery of global scholars relies on a comprehensive portrayal of their profiles. By constructing multidimensional scholar portraits, we can depict scholars' basic information, behavioral trajectories, research interests, and service needs from multiple dimensions. This comprehensive information can then be used for in-depth mining, collecting valuable intelligence and establishing a high-precision dynamic profile of editorial committee members. Furthermore, a comprehensive evaluation model can be established based on the above dimensions, providing an objective basis and support for decision-making for the evaluation of expert academic achievements, the precise positioning of scholars, and the recruitment of scholarly talent. The construction and management of scholar portraits first requires obtaining accurate and detailed basic information about the scholars.

[0003] Furthermore, active scholars often publish their research findings publicly, and individuals or institutions maintain and update their latest information. A scholar's homepage typically includes basic information such as their affiliation, professional title, position, work experience, educational background, and a biographical profile, providing information support for subsequent multi-dimensional scholar portraits. Furthermore, by integrating information such as work experience, educational background, and biographical profiles from a scholar's homepage, it is possible to predict the scholar's commonly used language.

[0004] Conventional homepage prediction relies on feature engineering to predict the appearance of names and institutions on webpages or domain names. However, this approach yields low accuracy in predicting scholar homepages. Therefore, a solution is urgently needed to improve the accuracy and efficiency of predicting scholar homepages and the language used. Summary of the Invention

[0005] The present application aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, the first purpose of this application is to propose a method for predicting a scholar's homepage and the language used. The method first uses search engine resources to retrieve scholar-related information and screens out web pages related to the scholar from the massive Internet; secondly, a feature extractor is used to design web page features and characterize the effective information of the web page; then, a semantic classifier is used to learn the semantic information contained in the web page text, and the semantic vector of the web page text description content is learned through the first training model. Combined with the web page feature vector, a classification prediction is made on whether the web page is the scholar's homepage; finally, through the scholar language intelligent question-answering module, a question is designed based on the language mastered by the scholar, and the second training model after fine-tuning is used to answer the question based on the text of the predicted homepage, thereby realizing the prediction of the language mastered by the scholar. This method makes full use of the retrieval capabilities of Internet search engines and the text semantic information of the retrieved web pages to predict the scholar's homepage and the language used, thereby improving the prediction effect of the scholar's homepage and the language used.

[0007] The second purpose of this application is to propose a prediction system for scholars' homepages and the languages ​​they use.

[0008] A third object of the present application is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a method for predicting a scholar's homepage and the language used, comprising the following steps:

[0010] Acquire multiple web pages related to the scholar and extract features of each of the web pages to generate a web page feature vector matrix corresponding to the multiple web pages;

[0011] Learning the semantic feature vector of the text in each of the web pages through a preset first training model to generate a semantic feature vector matrix corresponding to the plurality of web pages;

[0012] Predicting the scholar's homepage by combining the webpage feature vector matrix and the semantic feature vector matrix;

[0013] A question sentence in the language used by the scholar is designed, and based on the predicted text of the scholar's homepage, an answer to the question sentence is obtained through a preset second training model.

[0014] Optionally, in one embodiment of the present application, in the prediction method, obtaining multiple web pages related to the scholar and extracting features of each of the web pages includes:

[0015] Performing a joint search on the names and institutions of the scholars through a search engine to obtain the multiple web pages;

[0016] Obtaining the address, title, and summary of each of said web pages;

[0017] Access each web page according to its address and obtain the text in each web page;

[0018] Based on the obtained title, abstract and text of each of the web pages, features of each of the web pages are extracted by a feature extractor.

[0019] Optionally, in one embodiment of the present application, the characteristics of the web page include but are not limited to: the length of the web page title, the length of the web page abstract, the number of positive words contained, the number of negative words contained, whether the scholar's name is in the web page address, whether the institution name is in the web page address, whether the scholar's name is in the title, whether the institution name is in the title, whether the scholar's name is in the abstract, whether the institution name is in the abstract, whether the scholar's name is in the web page text, and whether the institution name is in the web page text.

[0020] Optionally, in one embodiment of the present application, the semantic feature vector of the text in each of the web pages is learned through a preset first training model to generate a semantic feature vector matrix corresponding to the multiple web pages, including: splitting the text of each of the web pages; preprocessing the text of each of the split web pages, the preprocessing including encoding and random masking; inputting the preprocessed text into the preset first training model to obtain a vector sequence corresponding to each of the web pages, wherein each vector in the vector sequence has the same dimension; converting the vector sequence corresponding to each of the web pages into a first vector of the same dimension through a long short-term memory artificial neural network LSTM, and combining the first vectors corresponding to each of the web pages to generate the semantic feature vector matrix.

[0021] Optionally, in one embodiment of the present application, the web page feature vector matrix and the semantic feature vector matrix are combined to predict the scholar's homepage, including: splicing the web page feature vector matrix and the semantic feature vector matrix to obtain the final vector of the multiple web pages; performing classification prediction on the final vector through a fully connected layer and converting the final vector into a two-dimensional vector; processing the two-dimensional vector through an activation function to obtain a classification score for each of the web pages, and determining the scholar's homepage from the multiple web pages based on the classification score of each of the web pages.

[0022] Optionally, in one embodiment of the present application, the method of obtaining an answer to the question based on the predicted text of the scholar's homepage through a preset second training model includes: fine-tuning the preset second training model; splicing the determined text of the scholar's homepage with the question in the language used by the scholar, and inputting the spliced ​​sentence into the fine-tuned second training model; returning the score of each word in the vocabulary of the second training model according to the spliced ​​sentence through the fine-tuned second training model, and taking the word with the highest score as the prediction result of the language used by the scholar.

[0023] Optionally, in one embodiment of the present application, fine-tuning the preset second training model includes:

[0024] S1: Generate question templates based on the scholar’s ​​name, institution, and language used;

[0025] S2: Obtain a training set, supplement the question template according to the information of scholars in the training set, and splice the text corresponding to the scholars in the training set and the supplemented question template;

[0026] S3: Reading the vocabulary of the second training model, encoding the spliced ​​question templates using the vocabulary, and inputting the generated coding list into the second training model;

[0027] S4: Generate a score for each word in the vocabulary using the second training model, and calculate a cross entropy loss based on the score of each word and the actual result;

[0028] S5: Adjust the parameters of the second training model according to the cross entropy loss, and repeat steps S2 to S4 until the second training model converges.

[0029] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a prediction system based on a scholar's homepage and the language used, including the following modules:

[0030] A webpage feature generation module, configured to obtain a plurality of webpages related to the scholar and extract features of each of the webpages, and generate a webpage feature vector matrix corresponding to the plurality of webpages;

[0031] A semantic feature generation module, configured to learn a semantic feature vector of text in each of the web pages using a preset first training model, and generate a semantic feature vector matrix corresponding to the plurality of web pages;

[0032] A homepage prediction module, configured to predict the scholar's homepage by combining the webpage feature vector matrix and the semantic feature vector matrix;

[0033] The language prediction module is used to design a question sentence in the language used by the scholar, and obtain an answer to the question sentence through a preset second training model based on the predicted text of the scholar's homepage.

[0034] Optionally, in one embodiment of the present application, in the prediction system, the webpage feature module is specifically configured to:

[0035] Performing a joint search on the names and institutions of the scholars through a search engine to obtain the multiple web pages;

[0036] Obtaining the address, title, and summary of each of said web pages;

[0037] Access each web page according to its address and obtain the text in each web page;

[0038] Based on the obtained title, abstract and text of each of the web pages, features of each of the web pages are extracted by a feature extractor.

[0039] Optionally, in one embodiment of the present application, the characteristics of the web page include but are not limited to: the length of the web page title, the length of the web page abstract, the number of positive words contained, the number of negative words contained, whether the scholar's name is in the web page address, whether the institution name is in the web page address, whether the scholar's name is in the title, whether the institution name is in the title, whether the scholar's name is in the abstract, whether the institution name is in the abstract, whether the scholar's name is in the web page text, and whether the institution name is in the web page text.

[0040] Optionally, in one embodiment of the present application, the semantic feature generation module is specifically used to: split the text of each of the web pages; preprocess the text of each of the split web pages, and the preprocessing includes encoding and random masking; input the preprocessed text into the preset first training model to obtain a vector sequence corresponding to each of the web pages, wherein each vector in the vector sequence has the same dimension; convert the vector sequence corresponding to each of the web pages into a first vector with the same dimension through a long short-term memory artificial neural network LSTM, and combine the first vectors corresponding to each of the web pages to generate the semantic feature vector matrix.

[0041] Optionally, in one embodiment of the present application, the homepage prediction module is specifically used to: splice the web page feature vector matrix with the semantic feature vector matrix to obtain the final vector of the multiple web pages; perform classification prediction on the final vector through a fully connected layer and convert the final vector into a two-dimensional vector; process the two-dimensional vector through an activation function to obtain the classification score of each web page, and determine the scholar's homepage from the multiple web pages based on the classification score of each web page.

[0042] Optionally, in one embodiment of the present application, the language prediction module is specifically used to: fine-tune the preset second training model; splice the text of the determined scholar's homepage with the question sentence in the language used by the scholar, and input the spliced ​​sentence into the fine-tuned second training model; return the score of each word in the vocabulary of the second training model according to the spliced ​​sentence through the fine-tuned second training model, and use the word with the highest score as the prediction result of the language used by the scholar.

[0043] Optionally, in one embodiment of the present application, in the prediction system, the language prediction module is specifically configured to:

[0044] S1: Generate question templates based on the scholar’s ​​name, institution, and language used;

[0045] S2: Obtain a training set, supplement the question template according to the information of scholars in the training set, and splice the text corresponding to the scholars in the training set and the supplemented question template;

[0046] S3: Reading the vocabulary of the second training model, encoding the spliced ​​question templates using the vocabulary, and inputting the generated coding list into the second training model;

[0047] S4: Generate a score for each word in the vocabulary using the second training model, and calculate a cross entropy loss based on the score of each word and the actual result;

[0048] S5: Adjust the parameters of the second training model according to the cross entropy loss, and repeat steps S2 to S4 until the second training model converges.

[0049] The technical solution provided by the embodiment of the present application brings at least the following beneficial effects: the present application first uses search engine resources to search for scholar-related information and screens out web pages related to scholars from the massive Internet; secondly, a feature extractor is used to design web page features and characterize the effective information of the web page; then, a semantic classifier is used to learn the semantic information contained in the web page text, and the semantic vector of the web page text description content is learned through the first training model, and the web page feature vector is combined to classify and predict whether the web page is the scholar's homepage; finally, through the scholar language intelligent question-answering module, questions are designed for the language mastered by the scholar, and the second training model after fine-tuning is used to answer the questions based on the text of the predicted homepage, thereby realizing the prediction of the language mastered by the scholar. This method makes full use of the retrieval capabilities of Internet search engines and the text semantic information of the retrieved web pages to predict the scholar's homepage and the language used, thereby improving the accuracy of the prediction of the scholar's homepage and the language used.

[0050] In order to implement the above embodiments, the third aspect of the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the method and system for predicting the scholar's homepage and the language used in the above embodiments.

[0051] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which

[0053] Figure 1 A flowchart of a method for predicting a scholar's homepage and the language used, proposed in an embodiment of the present application;

[0054] Figure 2 A flowchart of a specific method for predicting scholar homepages and the language used proposed in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of the structure of a system for predicting a scholar's homepage and the language used, proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0057] To address the challenges of related technologies, the applicant discovered that scholar homepages often contain valid textual information. Using semantic descriptions of textual content to determine whether it represents homepage content can aid in identifying scholar homepages. While language prediction typically relies on names, information contained within the text of internet webpages, such as publicly available resumes, can also be used to predict a scholar's language proficiency. Based on this, the applicant proposed a method and system for predicting a scholar's homepage and the language used.

[0058] The following describes a method and system for predicting a scholar's homepage and the language used, according to an embodiment of the present invention, with reference to the accompanying drawings.

[0059] Figure 1 This is a flowchart of a method for predicting a scholar's homepage and the language used, as proposed in an embodiment of the present application. Figure 1 The method comprises the following steps:

[0060] Step 101: Acquire multiple web pages related to the scholar and extract features of each web page to generate a web page feature vector matrix corresponding to the multiple web pages.

[0061] In one embodiment of the present application, a search engine is first used to jointly search the scholar's name and institution to obtain multiple web pages, and then the address, title and abstract of each web page are obtained. Then, each web page is accessed according to its address, and the text in each web page is obtained. Then, based on the obtained title, abstract and text of each web page, the features of each web page are extracted through a feature extractor.

[0062] Among them, the feature extractor is designed based on the characteristics of the homepage that distinguish it from other webpages, and is constructed by designing a homepage feature tag. In an embodiment of the present application, the features of the webpage extracted by the feature extractor include but are not limited to: the length of the webpage title, the length of the webpage abstract, the number of positive words contained, the number of negative words contained, whether the scholar's name is in the webpage address, whether the organization name is in the webpage address, whether the scholar's name is in the title, whether the organization name is in the title, whether the scholar's name is in the abstract, whether the organization name is in the abstract, whether the scholar's name is in the webpage text, and whether the organization name is in the webpage text. It should be noted that this application does not limit the features of the webpage obtained to only include the content in the above-mentioned exemplary description, that is, in the embodiment of the present application, other features representing the characteristics of the webpage can also be obtained, such as whether positive words are in the abstract, etc., which are not listed here one by one. Among them, the words that are conducive to being judged as being in the homepage are defined as positive words, otherwise they are negative words. For example, "academic", "homepage", "professor", "person" and so on are positive words, which are usually found on homepages; while "google", "pdf", "youtube", "news" and so on are negative words, which are usually not found on homepages.

[0063] It should be noted that this application uses a search engine to jointly search the names of scholars and their institutions, which can reduce the relevant information of other people with the same name and improve the accuracy of homepage predictions.

[0064] In one embodiment of the present application, after extracting the web page address in the search results for access and extracting and saving the text data in the corresponding web page, the extracted scholars' names and institutions are converted to lowercase English names, and the text is processed, including converting the text to lowercase and filtering out meaningless characters, thereby improving the standardization and accuracy of the extracted features.

[0065] Furthermore, the extracted features are combined to obtain a web page feature vector matrix X.

[0066] Step 102 : learning the semantic feature vector of the text in each web page through a preset first training model, and generating a semantic feature vector matrix corresponding to the plurality of web pages.

[0067] The first training model may be a semantic classifier, which is used to obtain text semantic information of the retrieved web page.

[0068] In one embodiment of the present application, during specific implementation, the text of each page can be split first, and then the text of each split web page can be preprocessed, wherein the preprocessing includes encoding and random masking, and then the preprocessed text is input into a preset first training model to obtain a vector sequence corresponding to each web page, and each vector in the vector sequence has the same dimension. Finally, the vector sequence corresponding to each web page is converted into a first vector with the same dimension through a long short-term memory artificial neural network LSTM, and the first vectors corresponding to each web page are combined to generate a semantic feature vector matrix.

[0069] Specifically, in order to solve the problem that the model cannot process long texts, the web page text process is first split. When splitting the text of each web page, the text in each web page can be split according to a fixed length. In this embodiment, the length is set to 300, and the last n words of each block of text are consistent with the first n words of the next block of text to strengthen the association between text blocks, where n>=0, and in this embodiment, n is 10. The split text blocks are then preprocessed, including encoding the words in the text, obtaining the encoding sequence of the words, and randomly masking the words in the text. Further, the preprocessed text blocks are input into the preset first training model to obtain the vector sequence {z1, z2, z3, ..., z k}, because the length of text on different web pages is different and the number of text blocks is different, the number of vectors z obtained is inconsistent. Among them, the dimension of each vector in the vector sequence is the same. Furthermore, the vector sequence {z1, z2, z3, ..., z k The input is then fed into a long short-term memory (LSTM) artificial neural network and converted into a first vector s of dimension 1*t. t is a fixed value preset to unify the dimensions; the specific value can be set as needed. Finally, the first vectors corresponding to each web page are combined to generate a semantic feature vector matrix S.

[0070] Step 103: predict the scholar's homepage by combining the webpage feature vector matrix and the semantic feature vector matrix.

[0071] In one embodiment of the present application, the webpage feature vector matrix X and the semantic feature vector matrix S are obtained in step 101 and step 102 respectively. In the specific implementation, the extracted feature matrix X is first concatenated with the semantic feature vector matrix S obtained by the semantic classifier to obtain the final vector matrix of the webpage. Then, a fully connected layer is used to perform classification prediction on the vector H, and the dimension of the final vector H is converted to two dimensions. Finally, the sigmoid activation function is used to process the dimensionality reduction result, that is, the two-dimensional vector is processed to obtain the web page classification score. Among them, the value of the first dimension in the processed two-dimensional vector represents the score of the web page not being the homepage, and the value of the second dimension represents the score of the web page being the homepage.

[0072] Furthermore, the web page links in the search web page list whose second dimension scores are greater than the first dimension scores are returned, and the web page links in the search web page list whose second dimension scores are greater than the first dimension scores are determined to be the predicted results of the scholar's homepage.

[0073] It should be noted that in one embodiment of the present application, the semantic classifier, i.e., the first training model, can be constructed by a pre-trained model, an LSTM layer, and a fully connected layer, that is, the present application can perform the above steps of generating the semantic feature vector matrix S and classification prediction through the semantic classifier.

[0074] Step 104: design a question sentence in the language used by the scholar, and obtain an answer to the question sentence through a preset second training model based on the predicted text of the scholar's homepage.

[0075] The second training model can be an intelligent question-answering module for scholar languages. The questions in the designed language can be question templates that include the scholar's name, affiliation, and the language to be predicted. For example, the questions in the scholar's language can be [NAME] at [ORG] can speak [MASK], where [MASK] is the language to be predicted.

[0076] In one embodiment of the present application, after determining the predicted result for a homepage, the text content of the predicted homepage is extracted. Questions are then formulated using the names and institutions of the scholars to be tested. These questions are then input into a second training model to generate answers in the language used by the scholars. To improve the accuracy of the predictions, the second training model can be fine-tuned before making the predictions. The specific method includes the following steps:

[0077] S1: Design a question template based on the input scholar's name, institution, and language: [NAME]at[ORG]can speak[MASK], where [NAME] is the scholar's name, [ORG] is the scholar's institution, and [MASK] is the answer you want to predict, that is, the language the scholar knows.

[0078] S2: Obtain the name, organization, language mastered, and related information of each scholar in the training set, process the text, convert it to lowercase, filter out meaningless characters, and then use the scholar's name and organization to supplement the question template, and splice the text of the homepage information corresponding to the scholar in the training set with the supplemented question template in the language used by the scholar. In one embodiment of the present application, after predicting the scholar's homepage in the above manner, the text of the scholar's homepage can be obtained, and after annotating the text of the homepage, the annotated data can be divided into a training set, a validation set, and a test set according to a preset ratio to facilitate training the model and subsequently verifying the prediction effect of the model through the data in the test set.

[0079] S3: Read the vocabulary of the second training model, use the vocabulary to encode the spliced ​​question template text, and input the generated encoding list into the second training model.

[0080] S4: Generate prediction scores for all words in the vocabulary using the second training model. Process the actual results into a 0-1 label sequence based on the predicted scores of the vocabulary, and then calculate the cross entropy loss between the score sequence and the actual results.

[0081] S5: Based on the size of the cross entropy loss, use the optimizer to adjust and optimize the parameters of the second training model, and repeat steps S2 to S4 until the second training model converges and fine-tuning is completed.

[0082] After fine-tuning, the language prediction is performed. Specifically, the text of the predicted homepage is extracted. Questions are then formulated using the names and institutions of the scholars to be tested, with the answers to be predicted marked within the questions. The text of the scholar's homepage is then concatenated with the concatenated language question. This concatenated sentence is then fed into the fine-tuned second training model. Based on this concatenated sentence, the second training model returns a score for each word in the vocabulary, with the word with the highest score being used as the predicted result for the scholar's language.

[0083] It should be noted that in one embodiment of the present application, the second training model can also return multiple prediction results. As an example, after the second training model returns the score of each word in the vocabulary, the score of each word is sorted in descending order according to the numerical value. According to the preset number of returns, the first three words are taken as possible prediction results of the language used by the scholar, so that the two words other than the word with the highest score can be used as a reference to improve the reliability of the prediction of the language used.

[0084] In summary, the prediction method of the scholar's homepage and the language used in the embodiment of the present application first uses search engine resources to retrieve scholar-related information and screen out web pages related to scholars from the massive Internet; secondly, the web page features are designed through a feature extractor to characterize the effective information of the web page; then the semantic information contained in the web page text is learned with the help of a semantic classifier, and the semantic vector of the web page text description content is learned through a first training model, and the web page feature vector is combined to classify and predict whether the web page is the scholar's homepage; finally, through the scholar language intelligent question-answering module, questions are designed for the language mastered by the scholar, and the second training model after fine-tuning is used to answer the questions based on the text of the predicted homepage, thereby realizing the prediction of the language mastered by the scholar. This method makes full use of the retrieval capability of the Internet search engine, trains the text semantic information of the retrieved web page, and improves the prediction effect of the scholar's homepage and language.

[0085] In order to more clearly illustrate the prediction method of the scholar's homepage and the language used in the embodiment of the present application, the following is combined with Figure 2 A specific example is used to illustrate:

[0086] like Figure 2 As shown, the steps of the method for predicting the scholar's homepage and the language used proposed in the embodiment of the present application are:

[0087] In the first step, we first obtain the search results of relevant information through the scholar's name and institution, and then use the feature extractor to obtain the features of each web page and obtain the web page feature vector matrix.

[0088] The specific steps are:

[0089] First, obtain the name of the scholar to be searched and the relevant information of the institution provided by the user.

[0090] Based on the scholar's name and institution information provided by the user, an internet search engine is used to retrieve relevant web pages containing the corresponding scholar's information. The retrieved webpage addresses and webpage summaries are returned. A feature extractor is then used to extract features from each webpage. Finally, the extracted features are combined to obtain the webpage feature vector matrix X.

[0091] The second step is to input the web page feature vector matrix into the semantic classifier to determine whether the web page is a scholar's homepage.

[0092] The specific steps are as follows: while using a feature extractor to extract the obvious features of the web page, the semantic information of the web page text is used to generate a web page semantic vector, and the web page feature vector and semantic vector are combined to classify and predict whether the web page is a scholar's homepage. The returned prediction is the web page address of the homepage.

[0093] The third step is to use the intelligent question-answering module to predict the language mastered by the scholars.

[0094] The specific steps are as follows: the language prediction module uses the rich information of the pre-trained model to set up an intelligent question-answering module for the scholar's language. By combining the homepage text with the generated questions, the pre-trained model is used to predict the answers to the questions and return the prediction results and scores of the language mastered by the scholar.

[0095] Finally, the prediction results are returned and displayed through an interface. This interface mainly receives the homepage prediction results and scholar language prediction results. In order to improve the interpretability of the model, certain explanations are given for the generation of the results so that users can understand and accept them.

[0096] In order to implement the above embodiment, the present application also proposes a system of scholars' homepages and languages ​​used.

[0097] Figure 3 This is a structural diagram of the system of scholars' homepages and languages ​​used proposed in the embodiment of this application.

[0098] like Figure 3 As shown, the system for the scholar's homepage and the language used includes a webpage feature module 100, a semantic feature module 200, a homepage prediction module 300, and a language prediction module 400.

[0099] The web page feature generation module 100 is used to obtain multiple web pages related to the scholar and extract features of each web page, and generate a web page feature vector matrix corresponding to the multiple web pages.

[0100] The semantic feature generation module 200 is configured to learn the semantic feature vector of the text in each web page through a preset first training model, and generate a semantic feature vector matrix corresponding to the plurality of web pages.

[0101] The homepage prediction module 300 is used to predict the scholar's homepage by combining the webpage feature vector matrix and the semantic feature vector matrix.

[0102] The language prediction module 400 is used to design a question sentence in the language used by the scholar, and obtain an answer to the question sentence through a preset second training model based on the predicted text of the scholar's homepage.

[0103] Optionally, in one embodiment of the present application, the web page feature generation module 100 is specifically used to: jointly search the scholar's name and institution through a search engine to obtain multiple web pages; obtain the address, title and abstract of each web page; access each web page according to the address and obtain the text in each web page; based on the obtained title, abstract and text of each web page, extract the features of each web page through a feature extractor.

[0104] In one embodiment of the present application, web page features include but are not limited to: the length of the web page title, the length of the web page abstract, the number of positive words contained, the number of negative words contained, whether the scholar's name is in the web page address, whether the institution name is in the web page address, whether the scholar's name is in the title, whether the institution name is in the title, whether the scholar's name is in the abstract, whether the institution name is in the abstract, whether the scholar's name is in the web page text, and whether the institution name is in the web page text.

[0105] Optionally, in one embodiment of the present application, the semantic feature generation module 200 is specifically used to: split the text of each web page; preprocess the text of each split web page, the preprocessing including encoding and random masking; input the preprocessed text into a preset first training model to obtain a vector sequence corresponding to each web page, wherein each vector in the vector sequence has the same dimension; convert the vector sequence corresponding to each web page into a first vector of the same dimension through a long short-term memory artificial neural network LSTM, and combine the first vectors corresponding to each web page to generate a semantic feature vector matrix.

[0106] Optionally, in one embodiment of the present application, the homepage prediction module 300 is specifically used to: splice the web page feature vector matrix and the semantic feature vector matrix to obtain the final vectors of multiple web pages; perform classification prediction on the final vector through a fully connected layer and convert the final vector into a two-dimensional vector; process the two-dimensional vector through an activation function to obtain the classification score of each web page, and determine the scholar's homepage from multiple web pages based on the classification score of each web page.

[0107] Optionally, in one embodiment of the present application, the language prediction module 400 is specifically used to: fine-tune the preset second training model; splice the text of the determined scholar's homepage with the question sentence in the language used by the scholar, and input the spliced ​​sentence into the fine-tuned second training model; return the score of each word in the vocabulary of the second training model according to the spliced ​​sentence through the fine-tuned second training model, and use the word with the highest score as the prediction result of the language used by the scholar.

[0108] Optionally, in one embodiment of the present application, the language prediction module 400 is further used to: generate a question template for the scholar's name, institution and language used; obtain a training set, supplement the question template according to the information of the scholar in the training set, and splice the text corresponding to the scholar in the training set and the supplemented question template; read the vocabulary of the second training model, encode the spliced ​​question template through the vocabulary, and input the generated coding list into the second training model; generate a score for each word in the vocabulary through the second training model, and calculate the cross-entropy loss based on the score of each word and the actual result; adjust the parameters of the second training model according to the cross-entropy loss, and repeat steps S2 to S4 until the second training model converges.

[0109] In summary, the prediction system of the scholar's homepage and the language used in the embodiment of the present application first uses search engine resources to retrieve scholar-related information and screen out web pages related to the scholar from the massive Internet; secondly, the web page features are designed through the feature extractor to characterize the effective information of the web page; then the semantic information contained in the web page text is learned by the semantic classifier, the semantic vector of the web page text description content is learned through the first training model, and the web page feature vector is combined to classify and predict whether the web page is the scholar's homepage; finally, the scholar language intelligent question-answering module is used to design questions for the language mastered by the scholar, and the second training model after fine-tuning is used to answer the questions based on the text of the predicted homepage, thereby realizing the prediction of the language mastered by the scholar. The system makes full use of the retrieval ability of the Internet search engine and the text semantic information of the retrieved web page to improve the prediction effect of the scholar's homepage and language.

[0110] In order to implement the above embodiments, the present invention also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the method for predicting a scholar's homepage and the language used as described in the embodiment of the first aspect of this application.

[0111] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, if schematic expressions of the above terms are used in multiple embodiments or examples, it does not mean that these embodiments or examples are the same. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0113] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0114] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0115] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0116] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0117] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0118] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for predicting a scholar's homepage and the language used, characterized in that: The following steps are involved: Conduct a joint search of the scholar's name and institution through a search engine to obtain multiple web pages; Obtaining the address, title, and summary of each of said web pages; Access each web page according to its address and obtain the text in each web page; Based on the obtained title, abstract and text of each of the web pages, extracting features of each of the web pages by a feature extractor to generate a web page feature vector matrix corresponding to the plurality of web pages; splitting the text of each of said web pages; Preprocessing the text of each split web page, wherein the preprocessing includes encoding and random masking; Inputting the preprocessed text into a preset first training model to obtain a vector sequence corresponding to each of the web pages, wherein each vector in the vector sequence has the same dimension; converting the vector sequence corresponding to each web page into a first vector of the same dimension using a long short-term memory artificial neural network (LSTM), and combining the first vectors corresponding to each web page to generate a semantic feature vector matrix; the first training model is constructed by a pre-trained model, a long short-term memory network (LSTM) layer, and a fully connected layer; Predicting the scholar's homepage by combining the webpage feature vector matrix and the semantic feature vector matrix; A question sentence in the language used by the scholar is designed, and based on the predicted text of the scholar's homepage, an answer to the question sentence is obtained through a preset second training model, wherein the second training model is a scholar language intelligent question-answering module.

2. The prediction method according to claim 1, characterized in that The characteristics of the web page include but are not limited to: the length of the web page title, the length of the web page abstract, the number of positive words contained, the number of negative words contained, whether the scholar's name is in the web page address, whether the institution name is in the web page address, whether the scholar's name is in the title, whether the institution name is in the title, whether the scholar's name is in the abstract, whether the institution name is in the abstract, whether the scholar's name is in the web page text, and whether the institution name is in the web page text.

3. The prediction method according to claim 1, wherein: The step of predicting the scholar's homepage by combining the webpage feature vector matrix and the semantic feature vector matrix includes: splicing the webpage feature vector matrix and the semantic feature vector matrix to obtain final vectors of the plurality of webpages; Performing classification prediction on the final vector through a fully connected layer and converting the final vector into a two-dimensional vector; The two-dimensional vector is processed by an activation function to obtain a classification score of each web page, and the homepage of the scholar is determined from the multiple web pages according to the classification score of each web page.

4. The prediction method according to claim 3, characterized in that The method of obtaining an answer to the question based on the predicted text of the scholar's homepage through a preset second training model includes: Fine-tuning the preset second training model; splicing the determined text of the scholar's homepage with the question sentence in the language used by the scholar, and inputting the spliced ​​sentence into the fine-tuned second training model; The fine-tuned second training model returns the score of each word in the vocabulary of the second training model according to the spliced ​​sentence, and uses the word with the highest score as the prediction result of the language used by the scholar.

5. The prediction method according to claim 4, characterized in that The fine-tuning of the preset second training model includes: S1: Generate question templates based on the scholar’s ​​name, institution, and language used; S2: Obtain a training set, supplement the question template according to the information of scholars in the training set, and splice the text corresponding to the scholars in the training set and the supplemented question template; S3: Reading the vocabulary of the second training model, encoding the spliced ​​question templates using the vocabulary, and inputting the generated coding list into the second training model; S4: Generate a score for each word in the vocabulary using the second training model, and calculate a cross entropy loss based on the score of each word and the actual result; S5: Adjust the parameters of the second training model according to the cross entropy loss, and repeat steps S2 to S4 until the second training model converges.

6. A system for predicting a scholar's homepage and the language used, characterized in that: include: a webpage feature generation module configured to perform a joint search of the scholar's name and institution through a search engine to obtain the plurality of webpages; obtain the address, title, and abstract of each of the webpages; access each of the webpages according to the address and obtain the text in each of the webpages; and extract features of each of the webpages through a feature extractor based on the obtained title, abstract, and text of each of the webpages to generate a webpage feature vector matrix corresponding to the plurality of webpages; A semantic feature generation module is configured to segment the text of each web page; preprocess the segmented text of each web page, the preprocessing including encoding and random masking; input the preprocessed text into a preset first training model to obtain a vector sequence corresponding to each web page, wherein each vector in the vector sequence has the same dimension; convert the vector sequence corresponding to each web page into a first vector of the same dimension using a long short-term memory artificial neural network (LSTM), and combine the first vectors corresponding to each web page to generate a semantic feature vector matrix; the first training model is constructed by a pretrained model, a long short-term memory network (LSTM) layer, and a fully connected layer; A homepage prediction module, configured to predict the scholar's homepage by combining the webpage feature vector matrix and the semantic feature vector matrix; A language prediction module is used to design questions in the language used by the scholar, and based on the predicted text of the scholar's homepage, obtain answers to the questions through a preset second training model, wherein the second training model is a scholar language intelligent question-answering module.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for predicting a scholar's homepage and the language used is implemented as described in any one of claims 1 to 5.

Images