Method and device for generating search engine selected abstracts

Abstract

The present invention discloses a method and device for generating a selected summary of a search engine. The method comprises: identifying a question-and-answer query term from search log data; obtaining search results corresponding to the question-and-answer query term; if the search results do not contain a document of a specified type, using the question-and-answer query term and the corresponding search results as input data, and outputting a labeled candidate answer corresponding to the question-and-answer query term according to a machine reading comprehension model; obtaining a labeled answer corresponding to the labeled candidate answer based on active learning, and using the labeled answer as a selected summary of the corresponding question-and-answer query term in the search engine. This technical solution provides a selected summary for the search engine that allows users to directly view the required content in the search results page by combining machine reading comprehension and active learning for the identified question-and-answer query term, taking into account both query cost and answer accuracy, and improving user experience and search quality.

Description

Technical Field

[0001] The present invention relates to the technical field of search engines, and in particular to a method and device for generating a search engine selected summary. Background Art

[0002] Traditional search engines perform operations such as query understanding, user intent analysis, and web page ranking based on user queries, and ultimately return highly relevant web pages to users as natural results. However, this approach has gradually failed to meet user needs. For example, in a real search scenario, a user may query "Why is sea water salty?" Under normal circumstances, the returned web pages require further selection by the user, and the user needs to click on the web page to obtain more complete content. If the user finds that he does not get a good answer after clicking on the web page, he may return to the original page and perform a second or even third screening and clicking process. Summary of the invention

[0003] In view of the above problems, the present invention is proposed to provide a method and device for generating search engine featured snippets that overcome the above problems or at least partially solve the above problems.

[0004] According to one aspect of the present invention, a method for generating a selected summary of a search engine is provided, comprising: identifying question-and-answer query terms from search log data; obtaining search results corresponding to the question-and-answer query terms; if the search results do not contain documents of a specified type, using the question-and-answer query terms and the corresponding search results as input data, and outputting annotated candidate answers corresponding to the question-and-answer query terms according to a machine reading comprehension model; obtaining annotated answers corresponding to the annotated candidate answers based on active learning, and using the annotated answers as the selected summary of the corresponding question-and-answer query terms in the search engine.

[0005] According to another aspect of the present invention, a device for generating selected summaries of a search engine is provided, comprising: an identification unit, adapted to identify question-and-answer query terms from search log data; a search unit, adapted to obtain search results corresponding to the question-and-answer query terms; a candidate unit, adapted to use the question-and-answer query terms and the corresponding search results as input data if the search results do not contain documents of a specified type, and output annotated candidate answers corresponding to the question-and-answer query terms according to a machine reading comprehension model; and a selected summary generation unit, adapted to obtain annotated answers corresponding to the annotated candidate answers based on active learning, and use the annotated answers as the selected summaries of the corresponding question-and-answer query terms in the search engine.

[0006] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes any of the above methods.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the one or more programs implement any of the methods described above.

[0008] From the above, it can be seen that the present invention identifies question-and-answer query words from search log data; obtains search results corresponding to the question-and-answer query words; if the search results do not contain documents of a specified type, uses the question-and-answer query words and the corresponding search results as input data, and outputs labeled candidate answers corresponding to the question-and-answer query words according to a machine reading comprehension model; obtains labeled answers corresponding to the labeled candidate answers based on active learning, and uses the labeled answers as the selected summaries of the corresponding question-and-answer query words in the search engine. For the identified question-and-answer query words, a technical solution is provided for the search engine through a combination of machine reading comprehension and active learning, in which a user can directly view the required content in the search results page, taking into account both the query cost and the answer accuracy, thereby improving the user experience and the search quality.

[0009] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0011] Figure 1 A schematic diagram showing a flow chart of a method for generating a search engine selected summary according to an embodiment of the present invention;

[0012] Figure 2 A schematic diagram of the structure of a device for generating a search engine selected summary according to an embodiment of the present invention is shown;

[0013] Figure 3 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown;

[0014] Figure 4 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention is shown;

[0015] Figure 5a A schematic diagram of a search result page obtained according to a question-answering query term in the prior art is shown;

[0016] Figure 5b A schematic diagram of a search result page obtained according to a question-answer query term according to an embodiment of the present invention is shown;

[0017] Figure 6a The question-answering query words of the fact-description type are shown;

[0018] Figure 6b The question-answering query terms of the fact-entity class are shown;

[0019] Figure 6c The question-answer type query words of the opinion-description type are shown;

[0020] Figure 6d The question-answering query terms of the opinion-entity query are shown;

[0021] Figure 7 A schematic diagram of a web page showing a document of a specified type;

[0022] Figure 8a A schematic diagram of a coarse-scale interface is shown;

[0023] Figure 8b A schematic diagram of a precision labeling interface is shown. DETAILED DESCRIPTION

[0024] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0025] The design idea of ​​the present invention is to provide a "selected summary" for question-and-answer query words, which is different from traditional search results, so that users can get more accurate answers in the search results page without clicking on the web page to jump. For example, the search engine can extract the definite answer to the user's question and put it in the first place of the returned results in a more eye-catching way, which can help users save a lot of time spent on getting the answer, thereby improving user experience and search quality. This process is the search engine helping users to select answers, so the present invention is called "selected summary".

[0026] It should be noted that, although the technical solution of the present invention is to provide selected summaries for search engines, the generation process of selected summaries can be independent of the existing processes of general search engines, and the adaptability to various search engines can be improved through loose coupling, which facilitates integration.

[0027] Figure 1 FIG. 2 is a flow chart showing a method for generating a search engine selected summary according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0028] Step 110: Identify question-answering query terms from the search log data.

[0029] Each user of a search engine can generate a large amount of search log data during daily searches, for example, which query terms the user searched for, which websites in the search results the user clicked after obtaining the search results, etc. Through statistics and research, it is found that for question-and-answer query terms, featured summaries are most helpful in improving the user's search experience. Therefore, the technical solution of the present invention mainly generates corresponding featured summaries for question-and-answer query terms.

[0030] Step 120, obtaining search results corresponding to the question-answering query words. This step can be achieved by calling an existing search engine interface or the like.

[0031] Step 130: If the search results do not contain documents of the specified type, the question-answer query and the corresponding search results are used as input data, and the labeled candidate answers corresponding to the question-answer query are output according to the machine reading comprehension model.

[0032] For documents of a specified type, since the type is known, content can be extracted through templates and other methods, but for other web pages, the difficulty lies in how to obtain information related to the answer required for question-answering query words. In this regard, the embodiment of the present invention adopts the MRC (Machine Reading Comprehension) method, using question-answering query words and corresponding search results as input data, and outputting the labeled candidate answers corresponding to the question-answering query words according to the machine reading comprehension model.

[0033] Since the accuracy of such answers may deviate from the actual needs of users, for example, during the verification process, it is found that the cold start effect of the model is not good, and the accuracy of the generated answers is not enough to be directly put online, then the embodiment of the present invention obtains the annotated answers corresponding to the annotated candidate answers based on active learning through step 140, and uses the annotated answers as the selected summaries of the corresponding question-and-answer query terms in the search engine. The traditional annotation method improves the model effect very slowly, and the annotation of the selected summaries has high requirements on the text management and knowledge reserves of the annotation personnel, making the annotation more difficult; therefore, the embodiment of the present invention further improves the accuracy through active learning.

[0034] visible, Figure 1 The method shown, for the identified question-and-answer query words, combines machine reading comprehension with active learning to provide the search engine with a selected summary of the content that users can directly view on the search results page, taking into account both the query cost and the accuracy of the answer, and improving the user experience and search quality.

[0035] Figure 5a A schematic diagram of a search result page obtained according to a question-answering query term in the prior art is shown; Figure 5b A schematic diagram of a search result page obtained according to a question-answering query word according to an embodiment of the present invention is shown. Figure 5b The first search result in the search results is a featured snippet, where users can get most of the answers without clicking into the web page, or they can click into the web page and get more information from the original page.

[0036] In one embodiment of the present invention, the above method includes: performing preset type processing on the search log data to extract query words that meet the requirements; using the query words as input data, and outputting question-answer query words according to the query word classification model.

[0037] Since there are a huge number of query terms searched by users, some of which may be in different forms and have different popularity, in order to take efficiency into consideration, we can selectively filter out query terms that meet the needs and use the query term classification model obtained by machine learning to classify them into question-and-answer query terms. For example, "Why is the sea blue" is a question-and-answer query term, while "1933" is not a question-and-answer query term, but a year query term.

[0038] In one embodiment of the present invention, the above method includes: sorting the query words in the search log data according to the page views PV, and extracting several query words from high to low in the sorting; and / or normalizing the query words to remove punctuation and / or spaces in the query words.

[0039] Since PV can reflect the popularity of a query word, for query words with low popularity, even if featured snippets are provided, the feedback of the effect is slow and not obvious. Therefore, in consideration of efficiency, the query words with high popularity can be extracted first. In addition, since the query words are not standardized, they can be normalized to remove punctuation and / or spaces, which can facilitate subsequent processing.

[0040] In one embodiment of the present invention, the above method also includes the following steps for preparing training data for the query term classification model: obtaining search log data generated within a preset time period as sample data; counting the total number of clicks on each query term in the sample data and the number of clicks on question-and-answer sites, and calculating the question-and-answer click ratio of each query term in the sample data within the preset time period; taking query terms whose question-and-answer click ratio is greater than a first threshold as positive examples, and taking query terms whose question-and-answer click ratio is lower than a second threshold as negative examples, to obtain training data.

[0041] Here, whether a query word is a question-and-answer query word is identified based on the user's clicks, and this is achieved through a query word classification model. An embodiment of the present invention provides an example of generating training data. In the process of making training data, two types of thresholds are preset, and the total number of clicks on each query word in the sample data and the number of clicks on question-and-answer sites are counted, and the proportion of question-and-answer clicks on each query word in the sample data within a preset time period is calculated. It is easy to understand that if a word is a question-and-answer query word, then after the user searches, there is a high probability that he will click on a question-and-answer site; and if it is not a question-and-answer query word, the user often will not click on a question-and-answer site, and positive and negative examples are generated accordingly. In a specific scenario, the first threshold can be 0.8, and the second threshold can be 0.1, and it has been verified that the obtained effect is better.

[0042] In one embodiment of the present invention, the above method also includes the following training step to obtain a query term classification model: dividing the training data into a training set, a validation set and a test set according to a preset ratio, and training based on the textCNN model to obtain a query term classification model.

[0043] For example, the training data is divided into a training set, a validation set, and a test set in a ratio of 7:2:1, and then input into the textCNN model for training. The textCNN model is a deep learning model that uses CNN to solve text classification problems and has a better effect on text classification.

[0044] In one embodiment of the present invention, the above method includes: taking the query words containing the specified words in the sample data as positive examples. For example, if a query word contains words such as "how" or "how", it can be basically determined to be a question-answering query word, so such a query word containing the specified words can be directly used as a positive example, or the query word containing the specified words can be removed from the negative examples counted according to the click ratio.

[0045] In one embodiment of the present invention, in the above method, the Q&A site is determined according to the URL pattern. For example, for the Q&A query term "how to make scrambled eggs with tomatoes", the corresponding answers can be found in different Q&A sites. Taking Zhihu, 360 Q&A and Baidu Knows as examples, the URLs of the answers are as follows:

[0046] Zhihu URL: https: / / www.zhihu.com / question / 19576438

[0047] Baidu Knows URL: https: / / zhidao.baidu.com / question / 572659771.html

[0048] 360 Q&A URL: https: / / wenda.so.com / q / 1532241908216013

[0049] In fact, through statistics, it is found that the URLs of question-and-answer sites all follow a certain URL pattern, such as the URL pattern of Zhihu is https: / / www.zhihu.com / question / …, the URL pattern of Baidu Knows is https: / / zhidao.baidu.com / question / …html, and the URL pattern of 360 Q&A is https: / / wenda.so.com / q / …. Therefore, the question-and-answer site can be determined based on the URL pattern.

[0050] In one embodiment of the present invention, the above method includes: preprocessing the identified question-answer query words, removing duplicate question-answer query words and removing question-answer query words that have been marked with answers. Such deduplication can reduce the repeated process of generating featured snippets and improve efficiency.

[0051] In one embodiment of the present invention, the above method includes: classifying the identified question-answering query words and filtering out the question-answering query words other than the preset type, so as to further filter out some question-answering query words that are not suitable for generating selected snippets.

[0052] In one embodiment of the present invention, the above method includes: classifying the identified question-answer query words according to at least one of a topic classification model, a query word classification model, and an answer classification model. As the name implies, the topic classification model can classify according to the topic of the question-answer query words, the query word classification model can classify according to the attributes of the query words themselves, and the answer classification model can classify according to the attributes of the answers to the question-answer query words.

[0053] In one embodiment of the present invention, in the above method, the subject type includes at least one of the following: mobile digital, life, games, education science, leisure and hobbies, culture and art, financial management, social livelihood, sports, and regions; the query term types include: fact type and / or opinion type; the answer type includes: description type and / or entity type.

[0054] The topic types shown above are the topic types that users often search for. The effect of generating selected snippets for such question-and-answer query words is more obvious. From the analysis of the properties of the question-and-answer query words themselves and the answers, we focus on four types of user queries. We believe that the selected snippets of these queries are most helpful in improving the user search experience. These four types of queries can be divided from two perspectives: according to the nature of the query, they can be divided into factual and opinion categories; according to the length characteristics of the answer, they can be divided into entity and description categories. For example, Figure 6a The question-answering query words of the fact-description type are shown; Figure 6b The question-answering query terms of the fact-entity class are shown; Figure 6c The question-answer type query words of the opinion-description type are shown; Figure 6d Question-answer type query terms for opinion-entity type queries are shown.

[0055] In one embodiment of the present invention, the above method includes: if the answer type of the question-and-answer query term is an entity class, calling a machine reading comprehension model that does not include a sorting algorithm to output multiple labeled candidate answers; otherwise, calling a complete machine reading comprehension model to output a labeled candidate answer.

[0056] Since the answers to entity-based question-and-answer queries are shorter, the annotation cost is lower, and multiple candidate answers can be output for marking. For other question-and-answer queries, a complete machine reading comprehension model including a ranking algorithm can be called to output the candidate answer with the highest score, which can reduce the annotation cost.

[0057] Before calling the machine reading comprehension model to generate annotated candidate answers, you can adjust the data format: This step mainly generates the format required for calling the machine reading comprehension model based on the data obtained in the previous step. A specific example is as follows:

[0058] Model input: The input of the model is a query and several documents, as follows:

[0059] {"query":"Why is the sky blue?",

[0060] "document":["Sunlight is composed of seven kinds of light: red, orange, yellow, green, cyan, blue and violet. Among these seven kinds of light, cyan, blue and violet have shorter wavelengths and are easily scattered by air molecules and dust. The scattering ability of atmospheric molecules and dust in the atmosphere for shorter wavelength blue light is much higher than that for other photons with longer wavelengths.",

[0061] "Shorter wavelengths of light, such as violet and blue, are more easily absorbed by air molecules than longer wavelengths of light, which are in the red, orange and yellow bands of the spectrum. The air molecules then radiate violet and blue light in different directions, saturating the sky."

[0062] … ]

[0064] }

[0065] Model output: The interval of the answer in the document is placed in the answer_spans field.

[0066] {"query":"Why is the sky blue?",

[0067] "document":["Sunlight is composed of seven kinds of light: red, orange, yellow, green, cyan, blue and violet. Among these seven kinds of light, cyan, blue and violet have shorter wavelengths and are easily scattered by air molecules and dust. The scattering ability of atmospheric molecules and dust in the atmosphere for shorter wavelength blue light is much higher than that for other photons with longer wavelengths.",

[0068] "Shorter wavelengths of light, such as violet and blue, are more easily absorbed by air molecules than longer wavelengths of light, which are in the red, orange and yellow bands of the spectrum. The air molecules then radiate violet and blue light in different directions, saturating the sky."

[0069] …

[0070] ],

[0071] "answer_spans": [44,77]

[0072] }

[0073] In one embodiment of the present invention, in the above method, the topic classification model is a text multi-classification model, and the method also includes the following steps for preparing training data for the topic classification model: obtaining search log data generated within a preset time period as sample data; counting the click ratio of each query word in the sample data on sites of different topic types; counting the page views of each query word in the sample data, and dividing high-frequency query words from the query words in the sample data according to the page views; and taking the topic type of the site with the highest click ratio of the high-frequency query word as the topic type of the corresponding high-frequency query word.

[0074] For example, when users search for "what is the configuration of iPhone XS", they often click on mobile phone and digital sites in the search results, such as Zhongguancun Online, and the subject type of the query term can be determined based on the click ratio. However, after testing, it is found that the effect is obvious for high-frequency words, that is, query words with high page views PV, while the effect is slightly worse for medium-frequency words and low-frequency words. Therefore, the present invention also provides examples of making training data for medium-frequency words and low-frequency words.

[0075] In one embodiment of the present invention, in the above method, the step of making training data for the topic classification model also includes: dividing medium-frequency query words from the query words in the sample data according to the page views; using the high-frequency query words with determined topic types as training sets, training a support vector machine SVM model, and determining the topic type of the medium-frequency query words according to the trained SVM model. SVM (Support Vector Machine) is a type of generalized linear classifier that performs binary classification on data in a supervised learning manner, and its decision boundary is the maximum margin hyperplane solved for the learning samples. It has been verified that this method has a higher classification accuracy for medium-frequency words.

[0076] In one embodiment of the present invention, in the above method, the step of making training data for the topic classification model also includes: dividing low-frequency query terms from the query terms of the sample data according to the page views; determining the topic type of the low-frequency query terms according to the syntactic dependency tree. Dependency Parsing (DP) reveals the syntactic structure by analyzing the dependency relationship between components within a language unit. Intuitively speaking, dependency syntactic analysis identifies grammatical components such as "subject, predicate, object", "attributive, adverbial, complement" in a sentence, and analyzes the relationship between each component. Usually, a tree structure is formed, and it has been verified that this method has a higher classification accuracy for low-frequency words.

[0077] In a specific embodiment, words with a daily average PV greater than 50 are regarded as high-frequency words, words with a daily average PV between 5 and 50 are regarded as medium-frequency words, and words with a daily average PV less than 5 are regarded as low-frequency words.

[0078] In one embodiment of the present invention, in the above method, both the query term classification model and the answer classification model are text multi-classification models; the query term classification model is trained based on the nature characteristics of the query term; and the answer classification model is trained based on the length characteristics of the answer. According to the above example, fact-based query terms have relatively uniform answers, while opinion-based query terms can have multiple answers; entity-based query terms have shorter answers, while description-based query terms have longer answers.

[0079] In one embodiment of the present invention, in the above method, identifying question-and-answer query words from search log data further includes: filtering the identified question-and-answer query words according to several specified dimensions to obtain filtered question-and-answer query words. Specifically, in one embodiment of the present invention, in the above method, filtering the identified question-and-answer query words according to several specified dimensions includes: filtering each specified dimension using a corresponding text classification model; the text classification model is obtained by training based on SVM and / or fastText. Different text classification models can be selected according to the type of dimension. The specified dimension can include query words that do not comply with relevant laws and regulations, sensitive query words, etc., and question-and-answer query words involving people and incoherent question-and-answer query words can also be filtered.

[0080] In one embodiment of the present invention, in the above method, obtaining search results corresponding to question and answer query terms includes: calling a search engine interface to obtain a first number of natural search results corresponding to the question and answer query terms according to the order of search results; adjusting the natural search results according to a preset algorithm, and selecting a second number of natural search results therefrom as search results corresponding to the corresponding question and answer query terms.

[0081] The technical solution of the present invention can achieve the acquisition of search results by calling the interface of an existing search engine, and the sorting order of search results by the search engine is not necessarily based on factors such as the most semantically relevant. Therefore, multiple natural search results with higher rankings can be screened out first, and then adjusted using a preset algorithm.

[0082] In addition, many search engines will rewrite the query terms, such as rewriting pinyin into Chinese characters, so the final generated featured snippet can also correspond to the rewritten words.

[0083] In one embodiment of the present invention, in the above method, adjusting the natural search results according to a preset algorithm includes at least one of the following: filtering out document sites that cannot obtain web page content from the natural search results; and improving the order of sites with a trustworthiness higher than a first preset value.

[0084] For document sites that cannot obtain web page content, if the content cannot be captured through subsequent steps, they will be filtered directly. In addition, for authoritative sites, the order can be improved.

[0085] In one embodiment of the present invention, in the above method, obtaining search results corresponding to question-and-answer query words further includes: filtering the question-and-answer query words according to the natural search results. Specifically, in one embodiment of the present invention, in the above method, filtering the question-and-answer query words according to the natural search results includes at least one of the following: filtering out question-and-answer query words containing application boxes in the natural search results; filtering out question-and-answer query words containing illegal words in the titles of the natural search results; filtering out question-and-answer query words that lack high-quality natural search results based on semantic matching.

[0086] Onebox is a mature way to display search results. It can give relatively ideal search results for search terms related to stocks, movies, TV series, weather, etc., so there is no need to generate selected snippets for these search terms. Since the machine reading comprehension model is needed for subsequent processing, if it is found based on semantic matching that there is a lack of high-quality natural search results, even if machine reading comprehension is performed, the desired answer cannot be obtained, and such question-and-answer search terms are also filtered. Illegal words can be question-and-answer search terms that do not comply with relevant laws and regulations.

[0087] In one embodiment of the present invention, in the above method, the method further includes: if the search results include a document of a specified type, directly extracting the annotation candidate answer from the document of the specified type. Specifically, in one embodiment of the present invention, in the above method, the document of the specified type is an HTML document containing several step description information, and directly extracting the annotation candidate answer from the document of the specified type includes: parsing the HTML document, extracting several step description information according to field matching, and obtaining the annotation candidate answer by splicing.

[0088] Figure 7 A schematic diagram of a web page of a document of a specified type is shown. Figure 7 As shown, it contains several steps, and these steps can be extracted to obtain the labeled candidate answers. One way is to use the Beautiful Soup library (Beautiful Soup is a Python package that parses HTML and XML documents and repairs documents with errors such as unclosed tags - such documents are often called tag soup) to parse HTML into objects for processing, and then match the relevant fields to find the content after the "step" in the web page, and then splice them into the labeled candidate answers for the question-and-answer query.

[0089] In one embodiment of the present invention, in the above method, taking question-and-answer query terms and corresponding search results as input data includes: calculating the semantic relevance score between the question-and-answer query terms and the web page titles of the search results according to a semantic matching model, and sorting the search results according to the semantic relevance score.

[0090] In order to further improve the efficiency and accuracy of answer generation, before actually generating answers, a semantic matching model can be used to semantically score question-and-answer query terms and web page titles of candidate documents, and re-rank them based on the scores, so that candidate documents with higher relevance are weighted and placed in a relatively forward position, so that the candidate documents contain possible correct answers as much as possible.

[0091] In one embodiment of the present invention, in the above method, the method also includes: if the calculated semantic relevance scores are all lower than the second preset value, then the output of the labeled candidate answers corresponding to the question and answer query terms according to the machine reading comprehension model and subsequent steps are not executed, and the selected summaries of the question and answer query terms are directly captured from the question and answer site.

[0092] If it is found that all candidate documents for question-and-answer query terms have low semantic matching scores with the question-and-answer query terms, it means that there are no high-quality documents for the question-and-answer query terms. If an answer is directly generated based on this, the answer is likely to be of poor quality. Therefore, the subsequent answer generation steps are not performed, and the "knowledge site crawling" process is entered.

[0093] In one embodiment of the present invention, in the above method, the method also includes the following steps of training a semantic matching model: obtaining question-answer query terms and web page title pairs containing positive examples and negative examples annotated, and constructing them as training data through a processor dictionary; based on the BERT pre-trained model and training data, fine-tune adjustment training is performed to obtain a semantic matching model.

[0094] 1) Obtain positive and negative examples through another annotation process, as follows:

[0095] Positive example: Why is the sky blue? Why is the sky blue?

[0096] Negative example: Why is the sky blue and why is the sea salty?

[0097] 2) Construct the processor dictionary and the data processing flow to form the data format required by the model, as follows:

[0098] Positive example: 1\tWhy is the sky blue\tWhy is the sky blue?

[0099] Negative example: 0\tWhy is the sky blue\tWhy is the sea salty

[0100] 3) Fine-tune: Run run_classifier.py to train the model.

[0101] In one embodiment of the present invention, in the above method, the method further includes: for question-and-answer query words whose PV is lower than a third preset value, directly crawling selected snippets from question-and-answer sites.

[0102] For question-and-answer search terms with low PV, since there are fewer searchers, crawling selected snippets from question-and-answer sites can save resources.

[0103] In one embodiment of the present invention, in the above method, the method further includes: generalizing question-answer type query words to obtain semantically similar query words; and marking the answers as selected summaries of semantically similar query words in the search engine.

[0104] On the one hand, semantically similar queries have the same answers, so there is no need to generate new answers. On the other hand, in order to improve the recall rate of selected snippets on search engines, we need to generalize the query. So there is "coverage expansion", which is based on the accumulation of a certain amount of high-quality query-answer data (query-answer library, QA library).

[0105] In one embodiment of the present invention, in the above method, question-and-answer query words are generalized to obtain semantically similar query words, including: based on the query word display situation, user click behavior and co-click behavior, mining candidate query words corresponding to the question-and-answer query words; calculating the semantic relevance score of the question-and-answer query words and each candidate query word according to the semantic matching model, and taking the candidate query words with a semantic relevance score higher than a fourth preset value as semantically similar query words.

[0106] For example, from massive search logs, using information such as query display, user click behavior, and co-click behavior, a group of new queries with similar semantics to the queries in the QA library are mined as candidate sets for expanding coverage. After that, a Bert-based semantic matching model can be called (the training process can refer to the aforementioned embodiment) to score the query-query pair. The higher the score, the closer the semantics of the two are. For example, a query with a score higher than 0.9 is selected as a synonymous query, and coverage is successfully expanded.

[0107] In one embodiment of the present invention, in the above method, generalizing the question and answer query words to obtain semantically similar query words includes: representing the query words by vectors, and determining the candidate query words corresponding to the question and answer query words by calculating the cosine similarity of each vector; calculating the semantic relevance score between the question and answer query words and each candidate query word according to a semantic matching model, and if the highest semantic relevance score is greater than a fifth preset value, then using the candidate query word corresponding to the highest semantic relevance score as a semantically similar query word.

[0108] This method can be understood as a method based entirely on semantics, called retrieval-based matching. When determining whether a new query can be triggered, it is necessary to find the query with the closest semantics to it, which is divided into two stages: recall and matching. In the recall stage, the main purpose is to initially screen out the candidate set. For example, the sentence2vector (sen2vec) model is used to map the query into a vector representation, and then the cosine similarity between the vectors is calculated to obtain the 200 queries with the highest semantic similarity from the library. In the matching stage, these 200 queries and the new query are respectively combined into query-query pairs, and the semantic matching model is called to score, so as to select the query pair with the highest score value and above a certain threshold as the final synonym pair.

[0109] A specific example is as follows:

[0110] recall:

[0111] 1) Vector representation of query: Call the sen2vec model and represent the input query as a vector;

[0112] 2) Calculate cosine similarity: Use the cosine value of the angle between two vectors in the vector space as a measure of the difference between two individuals.

[0113] 3) Sorting: Sort by cosine similarity and select the top 200 query pairs as the expansion trigger candidate set;

[0114] match:

[0115] 1) The semantic matching model is used to filter out the query pairs with the highest semantic scores from the 200 query pairs in the candidate set;

[0116] 2) Determine whether the score of the query pair is higher than a certain threshold. If it is higher, it is used as the final expansion trigger result.

[0117] In one embodiment of the present invention, in the above method, obtaining the labeled answer corresponding to the labeled candidate answer based on active learning includes: providing a fine labeling interface and a coarse labeling interface; displaying a unique labeled candidate answer through the coarse labeling interface, receiving the returned correctness evaluation information, and determining the labeled answer based on the correctness evaluation information; and displaying multiple labeled candidate answers through the fine labeling interface and receiving the returned labeled answers.

[0118] Through verification, it was found that whether the answers were extracted from the specified documents or generated by the machine reading comprehension model, their accuracy was far below 95%, which could not meet the requirements for being put online on the search engine. Therefore, the generated answers were annotated through active learning to improve the accuracy. In order to maximize the benefits of annotation, reduce the cost of annotation, and speed up the improvement of model effects, the solution uses the idea of ​​active learning and sets two annotation tasks at the output end: coarse annotation and fine annotation.

[0119] First, we use rough marking to determine whether the question and answer are correct or not. We need to determine whether the answer can answer the question well and classify it into the following four categories: correct answer, wrong answer, wrong query, and undetermined. If the marking result is that the answer is correct, the data can be directly put online; if the answer is wrong, it means that this is an error made by the machine reading comprehension model, and the model input data of this query is sent to the precise marking for marking, that is, the annotator needs to circle the best answer from a query and several documents. If the appropriate answer cannot be selected from these documents, the annotator can also directly fill in the best answer. On the one hand, the data marked with precise marking can be directly put online, and on the other hand, it can also be used as a training sample for the machine reading comprehension model to correct the model's errors and improve the model effect at the fastest speed.

[0120] The above combination of fine and rough annotation not only takes into account the annotation cost, but also improves the online output rate, and can also help the model to be optimized and iterated quickly. The following are the annotation pages of the annotation platform for rough and fine annotation.

[0121] Figure 8a A schematic diagram of a rough interface is shown. Figure 8b A schematic diagram of a precision labeling interface is shown.

[0122] In one embodiment of the present invention, in the above method, using the marked answer as a selected summary of the corresponding question-answer query in the search engine includes: saving the selected summary in XML format. The XML format data can be uploaded to a stable interface and finally configured in the database of the search engine.

[0123] Figure 2FIG. 2 shows a schematic diagram of a structure of a device for generating a search engine selected summary according to an embodiment of the present invention. Figure 2 As shown, the search engine selected summary generation device 200 includes:

[0124] The identification unit 210 is adapted to identify question-answering query words from the search log data.

[0125] Each user of a search engine can generate a large amount of search log data during daily searches, for example, which query terms the user searched for, which websites in the search results the user clicked after obtaining the search results, etc. Through statistics and research, it is found that for question-and-answer query terms, featured summaries are most helpful in improving the user's search experience. Therefore, the technical solution of the present invention mainly generates corresponding featured summaries for question-and-answer query terms.

[0126] The search unit 220 is adapted to obtain search results corresponding to question-answering query words, which can be implemented by calling existing search engine interfaces and the like.

[0127] The candidate unit 230 is adapted to take the question-and-answer query term and the corresponding search result as input data if the search result does not contain a document of a specified type, and output a labeled candidate answer corresponding to the question-and-answer query term according to the machine reading comprehension model.

[0128] For documents of a specified type, since the type is known, content can be extracted through templates and other methods, but for other web pages, the difficulty lies in how to obtain information related to the answer required for question-answering query words. In this regard, the embodiment of the present invention adopts the MRC (Machine Reading Comprehension) method, using question-answering query words and corresponding search results as input data, and outputting the labeled candidate answers corresponding to the question-answering query words according to the machine reading comprehension model.

[0129] Since the accuracy of such answers may deviate from the actual needs of users, for example, during the verification process, it is found that the cold start effect of the model is not good, and the accuracy of the generated answers is not enough to be directly put online, then the embodiment of the present invention is suitable for obtaining the annotated answers corresponding to the annotated candidate answers based on active learning through the selected summary generation unit 240, and using the annotated answers as the selected summaries of the corresponding question-and-answer query terms in the search engine. The traditional annotation method improves the model effect very slowly, and the annotation of the selected summary has high requirements on the text management and knowledge reserve of the annotation personnel, which makes the annotation more difficult; therefore, the embodiment of the present invention further improves the accuracy through active learning.

[0130] visible, Figure 2The device shown, for identified question-and-answer query words, combines machine reading comprehension with active learning to provide the search engine with a selected summary of the content that users can directly view on the search results page, taking into account both query cost and answer accuracy, and improving user experience and search quality.

[0131] In one embodiment of the present invention, in the above-mentioned device, the identification unit 210 is suitable for performing preset type processing on the search log data to extract query words that meet the requirements; taking the query words as input data, and outputting question-answer query words according to the query word classification model.

[0132] In one embodiment of the present invention, in the above-mentioned device, the identification unit 210 is suitable for sorting the query words in the search log data according to the page views PV, and extracting several query words from high to low in the sorting; and / or normalizing the query words to remove punctuation and / or spaces in the query words.

[0133] In one embodiment of the present invention, the above-mentioned device also includes: a training unit, which is suitable for obtaining search log data generated within a preset time period as sample data; counting the total number of clicks on each query word in the sample data and the number of clicks on question-and-answer sites, and calculating the question-and-answer click ratio of each query word in the sample data within the preset time period; taking query words whose question-and-answer click ratio is greater than a first threshold as positive examples, and taking query words whose question-and-answer click ratio is lower than a second threshold as negative examples, to obtain training data.

[0134] In one embodiment of the present invention, in the above-mentioned device, the training unit is suitable for dividing the training data into a training set, a validation set and a test set according to a preset ratio, and performing training based on the textCNN model to obtain a query term classification model.

[0135] In one embodiment of the present invention, in the above-mentioned apparatus, the training unit is adapted to take the query words containing the designated words in the sample data as positive examples.

[0136] In one embodiment of the present invention, in the above-mentioned device, the question-and-answer site is determined according to a URL pattern.

[0137] In one embodiment of the present invention, in the above-mentioned device, the identification unit 210 is adapted to pre-process the identified question-and-answer query words, remove duplicate question-and-answer query words, and remove question-and-answer query words for which marked answers have been obtained.

[0138] In one embodiment of the present invention, in the above device, the identification unit 210 is adapted to classify the identified question-answer query words and filter out the question-answer query words other than the preset types.

[0139] In one embodiment of the present invention, in the above-mentioned device, the identification unit 210 is adapted to classify the identified question-and-answer query terms according to at least one of a topic classification model, a query term classification model, and an answer classification model.

[0140] In one embodiment of the present invention, in the above-mentioned device, the subject type includes at least one of the following: mobile digital, life, games, education science, leisure and hobbies, culture and art, financial management, social livelihood, sports, and regions; the query term types include: fact type and / or opinion type; the answer type includes: description type and / or entity type.

[0141] In one embodiment of the present invention, in the above-mentioned device, the candidate unit 230 is suitable for calling a machine reading comprehension model that does not include a sorting algorithm to output multiple labeled candidate answers if the answer type of the question-and-answer query term is an entity class; otherwise, calling a complete machine reading comprehension model to output a labeled candidate answer.

[0142] In one embodiment of the present invention, in the above-mentioned device, the topic classification model is a text multi-classification model, and the device also includes: a training unit, suitable for obtaining search log data generated within a preset time period as sample data; counting the click ratio of each query word in the sample data on sites of different topic types; counting the page views of each query word in the sample data, and dividing high-frequency query words from the query words in the sample data according to the page views; and taking the topic type of the site with the highest click ratio of the high-frequency query word as the topic type of the corresponding high-frequency query word.

[0143] In one embodiment of the present invention, in the above-mentioned device, the training unit is suitable for dividing medium-frequency query words from the query words in the sample data according to the page views; using the high-frequency query words with determined subject types as training sets, training a support vector machine SVM model, and determining the subject types of the medium-frequency query words according to the trained SVM model.

[0144] In one embodiment of the present invention, in the above-mentioned device, the training unit is adapted to classify low-frequency query words from query words in sample data according to page views; and determine the topic type of the low-frequency query words according to the syntactic dependency tree.

[0145] In one embodiment of the present invention, in the above-mentioned device, the query term classification model and the answer classification model are both text multi-classification models; the query term classification model is trained based on the nature characteristics of the query term; and the answer classification model is trained based on the length characteristics of the answer.

[0146] In one embodiment of the present invention, in the above-mentioned device, the identification unit 210 is adapted to filter the identified question-answer query words according to a number of specified dimensions to obtain filtered question-answer query words.

[0147] In one embodiment of the present invention, in the above-mentioned device, the recognition unit 210 is suitable for filtering each designated dimension using a corresponding text classification model; the text classification model is obtained by training based on SVM and / or fastText respectively.

[0148] In one embodiment of the present invention, in the above-mentioned device, the search unit 220 is suitable for calling the search engine interface to obtain a first number of natural search results corresponding to question-and-answer query terms according to the order of search results; the natural search results are adjusted according to a preset algorithm, and a second number of natural search results are selected therefrom as the search results corresponding to the corresponding question-and-answer query terms.

[0149] In one embodiment of the present invention, in the above-mentioned device, the search unit 220 is adapted to filter out document-type sites that cannot obtain webpage content from natural search results; and to improve the order of sites whose trustworthiness is higher than a first preset value.

[0150] In one embodiment of the present invention, in the above-mentioned device, the search unit 220 is adapted to filter the question-and-answer query terms according to the natural search results.

[0151] In one embodiment of the present invention, in the above-mentioned device, the search unit 220 is suitable for filtering out question-and-answer query words containing application boxes in natural search results; and / or, filtering out question-and-answer query words containing illegal words in the titles of natural search results; and / or, based on semantic matching, filtering out question-and-answer query words that lack high-quality natural search results.

[0152] In one embodiment of the present invention, in the above-mentioned apparatus, the candidate unit 230 is further adapted to directly extract the marked candidate answers from the documents of the specified type if the search results include documents of the specified type.

[0153] In one embodiment of the present invention, in the above-mentioned device, the document of the specified type is an HTML document containing several step description information, and the candidate unit 230 is suitable for parsing the HTML document, extracting several step description information according to field matching, and obtaining the labeled candidate answers by splicing.

[0154] In one embodiment of the present invention, in the above-mentioned device, the candidate unit 230 is adapted to calculate the semantic relevance score between the question-answer query term and the web page title of the search result according to the semantic matching model, and sort the search results according to the semantic relevance score.

[0155] In one embodiment of the present invention, in the above-mentioned device, the candidate unit 230 is suitable for not executing the output of the labeled candidate answers corresponding to the question-and-answer query terms according to the machine reading comprehension model and the subsequent steps if the calculated semantic relevance scores are all lower than the second preset value; the selected summary unit is suitable for directly grabbing the selected summary of the question-and-answer query terms from the question-and-answer site.

[0156] In one embodiment of the present invention, the above-mentioned device also includes: a training unit, suitable for obtaining question-answer query terms and web page title pairs containing positive examples and negative examples marked, and constructing them into training data through a processor dictionary; based on the BERT pre-trained model and training data, fine-tune adjustment training is performed to obtain a semantic matching model.

[0157] In one embodiment of the present invention, in the above-mentioned device, the selected summary unit is suitable for directly fetching selected summaries from question-and-answer websites for question-and-answer query words whose PV is lower than a third preset value.

[0158] In one embodiment of the present invention, in the above-mentioned device, the selected summary unit is further suitable for generalizing question-answer type query words to obtain semantically similar query words; and marking the answers as selected summaries of semantically similar query words in the search engine.

[0159] In one embodiment of the present invention, in the above-mentioned device, the selected summary unit is suitable for mining candidate query words corresponding to question-and-answer query words based on query word display, user click behavior and co-click behavior; calculating the semantic relevance score between the question-and-answer query words and each candidate query word according to the semantic matching model, and taking the candidate query words with a semantic relevance score higher than a fourth preset value as semantically similar query words.

[0160] In one embodiment of the present invention, in the above-mentioned device, the selected summary unit is suitable for representing the query term by vector, and determining the candidate query term corresponding to the question and answer query term by calculating the cosine similarity of each vector; calculating the semantic relevance score of the question and answer query term and each candidate query term according to the semantic matching model, and if the highest semantic relevance score is greater than the fifth preset value, the candidate query term corresponding to the highest semantic relevance score is used as a query term with similar semantics.

[0161] In one embodiment of the present invention, in the above-mentioned device, the selected summary unit is suitable for providing a fine mark interface and a rough mark interface; displaying a unique marking candidate answer through the rough mark interface, receiving the returned correctness evaluation information, and determining the marking answer according to the correctness evaluation information; and displaying multiple marking candidate answers through the fine mark interface, and receiving the returned marking answers.

[0162] In one embodiment of the present invention, in the above device, the selected summary unit is adapted to save the selected summary in XML format.

[0163] It should be noted that the specific implementation of the above-mentioned device embodiments can be carried out with reference to the specific implementation of the above-mentioned corresponding method embodiments, which will not be repeated here.

[0164] To summarize, the present invention identifies question-and-answer query words from search log data; obtains search results corresponding to the question-and-answer query words; if the search results do not contain documents of a specified type, uses the question-and-answer query words and the corresponding search results as input data, and outputs labeled candidate answers corresponding to the question-and-answer query words according to a machine reading comprehension model; obtains labeled answers corresponding to the labeled candidate answers based on active learning, and uses the labeled answers as a selected summary of the corresponding question-and-answer query words in the search engine. For the identified question-and-answer query words, a technical solution is provided for the search engine through a combination of machine reading comprehension and active learning, in which a user can directly view the required content in the search results page, taking into account both query cost and answer accuracy, thereby improving user experience and search quality.

[0165] It should be noted that:

[0166] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general-purpose devices can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific language is for the purpose of disclosing the best mode of the present invention.

[0167] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0168] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0169] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0170] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0171] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the device for generating the selected summary of the search engine according to an embodiment of the present invention. The present invention may also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0172] For example, Figure 3A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. The electronic device 300 includes a processor 310 and a memory 320 arranged to store computer executable instructions (computer readable program code). The memory 320 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. The memory 320 has a storage space 330 for storing a computer readable program code 331 for executing any method step in the above method. For example, the storage space 330 for storing computer readable program code may include individual computer readable program codes 331 for implementing various steps in the above method respectively. The computer readable program code 331 can be read from or written into one or more computer program products. These computer program products include program code carriers such as a hard disk, a compact disk (CD), a memory card or a floppy disk. Such a computer program product is typically, for example Figure 4 The computer-readable storage medium. Figure 4 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention is shown. The computer-readable storage medium 400 stores a computer-readable program code 331 for executing the method steps according to the present invention, which can be read by the processor 310 of the electronic device 300. When the computer-readable program code 331 is executed by the electronic device 300, the electronic device 300 executes the various steps in the method described above. Specifically, the computer-readable program code 331 stored in the computer-readable storage medium can execute the method shown in any of the above embodiments. The computer-readable program code 331 can be compressed in an appropriate form.

[0173] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0174] The embodiment of the present invention discloses A1, a method for generating a search engine selected summary, comprising:

[0175] Identify question-answering query terms from search log data;

[0176] Obtain search results corresponding to the question-and-answer query term;

[0177] If the search results do not contain documents of the specified type, the question-and-answer query and the corresponding search results are used as input data, and a labeled candidate answer corresponding to the question-and-answer query is output according to a machine reading comprehension model;

[0178] Based on active learning, an annotated answer corresponding to the annotated candidate answer is obtained, and the annotated answer is used as a selected summary of the corresponding question-and-answer query term in the search engine.

[0179] A2. The method as described in A1, wherein identifying question-answering query words from the search log data comprises:

[0180] Perform preset processing on search log data to extract query words that meet the requirements;

[0181] The query words are used as input data, and question-answering query words are output according to the query word classification model.

[0182] A3. The method as described in A2, wherein the processing of the search log data by a preset type to extract the query words that meet the requirements comprises:

[0183] Sort the query words in the search log data by page views PV, and extract several query words from high to low in the order;

[0184] and / or,

[0185] Normalize the query words and remove punctuation and / or spaces in the query words.

[0186] A4. The method as described in A2, wherein the method further comprises the following step of preparing training data for the query term classification model:

[0187] Obtaining search log data generated within a preset time period as sample data;

[0188] Counting the total number of clicks on each query word in the sample data and the number of clicks on question-and-answer sites, and calculating the percentage of question-and-answer clicks on each query word in the sample data within a preset time period;

[0189] The query words whose question-answer click ratio is greater than the first threshold are taken as positive examples, and the query words whose question-answer click ratio is lower than the second threshold are taken as negative examples to obtain training data.

[0190] A5. The method as described in A4, wherein the method further comprises the following steps of training to obtain the query term classification model:

[0191] The training data is divided into a training set, a validation set and a test set according to a preset ratio, and is trained based on a textCNN model to obtain the query term classification model.

[0192] A6. The method as described in A4, wherein the step of preparing the training data of the query term classification model further comprises:

[0193] The query words containing the designated words in the sample data are taken as positive examples.

[0194] A7. The method as described in A4, wherein the question-and-answer site is determined based on a URL pattern.

[0195] A8. The method as described in A1, wherein the step of identifying question-answering query terms from the search log data further comprises:

[0196] The identified question-and-answer query words are preprocessed to remove duplicate question-and-answer query words and question-and-answer query words that have been annotated with answers.

[0197] A9. The method as described in A1, wherein the step of identifying question-answering query terms from the search log data further comprises:

[0198] The identified question-and-answer query words are classified and the question-and-answer query words other than the preset types are filtered out.

[0199] A10. The method as described in A9, wherein classifying the identified question-answer query terms comprises:

[0200] The identified question-and-answer query terms are classified according to at least one of a topic classification model, a query term classification model, and an answer classification model.

[0201] A11. The method as described in A10, wherein:

[0202] The subject types include at least one of the following: mobile digital, life, games, education science, leisure and hobbies, culture and art, financial management, social livelihood, sports, and regions;

[0203] The query word types include: factual and / or opinion;

[0204] Answer types include: description class and / or entity class.

[0205] A12. The method as described in A11, wherein the step of taking the question-answer query and the corresponding search results as input data and outputting the labeled candidate answers corresponding to the question-answer query according to the machine reading comprehension model comprises:

[0206] If the answer type of the question-and-answer query term is an entity class, a machine reading comprehension model that does not include a sorting algorithm is called to output multiple labeled candidate answers; otherwise, a complete machine reading comprehension model is called to output one labeled candidate answer.

[0207] A13. The method as described in A10, wherein the topic classification model is a text multi-classification model, and the method further comprises the following step of preparing training data for the topic classification model:

[0208] Obtaining search log data generated within a preset time period as sample data;

[0209] Counting the click ratio of each query word in the sample data on sites of different subject types;

[0210] Counting the page views of each query word in the sample data, and classifying high-frequency query words from the query words in the sample data according to the page views;

[0211] The subject type of the site with the highest click ratio for the high-frequency query word is used as the subject type of the corresponding high-frequency query word.

[0212] A14. The method as described in A13, wherein the step of preparing the training data of the topic classification model further comprises:

[0213] Dividing the query words of the sample data into medium-frequency query words according to the page views;

[0214] The high-frequency query words with determined subject types are used as training sets to train a support vector machine (SVM) model, and the subject types of the medium-frequency query words are determined based on the trained SVM model.

[0215] A15. The method as described in A13, wherein the step of preparing the training data of the topic classification model further comprises:

[0216] Classify low-frequency query words from the query words in the sample data according to the page views;

[0217] The topic type of the low-frequency query word is determined according to the syntactic dependency tree.

[0218] A16. The method as described in A10, wherein the query term classification model and the answer classification model are both text multi-classification models;

[0219] The query word classification model is obtained by training based on the property characteristics of the query words;

[0220] The answer classification model is trained based on the length feature of the answer.

[0221] A17. The method as described in A1, wherein the step of identifying question-answering query terms from the search log data further comprises:

[0222] The identified question-and-answer query words are filtered according to a number of specified dimensions to obtain filtered question-and-answer query words.

[0223] A18. The method as described in A17, wherein filtering the identified question-answer query words according to several specified dimensions comprises:

[0224] For each designated dimension, a corresponding text classification model is used for filtering; the text classification model is obtained by training based on SVM and / or fastText.

[0225] A19. The method as described in A1, wherein obtaining search results corresponding to the question-and-answer query term comprises:

[0226] Calling a search engine interface to obtain a first number of natural search results corresponding to the question-and-answer query term according to the search result order;

[0227] The natural search results are adjusted according to a preset algorithm, and a second number of natural search results are selected therefrom as search results corresponding to the corresponding question-and-answer query terms.

[0228] A20. The method as described in A19, wherein the adjusting the natural search results according to a preset algorithm comprises at least one of the following:

[0229] Filter out document sites that cannot obtain web page content from the natural search results;

[0230] The order of sites whose trustworthiness is higher than the first preset value is promoted.

[0231] A21. The method as described in A19, wherein the step of obtaining search results corresponding to the question-and-answer query term further comprises:

[0232] The question-and-answer query terms are filtered according to the natural search results.

[0233] A22. The method as described in A21, wherein filtering the question-answer query terms according to the natural search results comprises at least one of the following:

[0234] Filter out Q&A search terms that contain app boxes in natural search results;

[0235] Filter out Q&A search terms that contain illegal words in the titles of natural search results;

[0236] Based on semantic matching, filter out question-and-answer search terms that lack high-quality natural search results.

[0237] A23. The method as described in A1, wherein the method further comprises:

[0238] If the search results include documents of a specified type, the labeled candidate answers are directly extracted from the documents of the specified type.

[0239] A24. The method as described in A23, wherein the document of the specified type is an HTML document containing a plurality of step description information, and the step of directly extracting the annotation candidate answer from the document of the specified type comprises:

[0240] The HTML document is parsed, the plurality of step description information are extracted according to field matching, and the labeled candidate answers are obtained by splicing.

[0241] A25. The method as described in A1, wherein the step of using the question-and-answer query term and the corresponding search result as input data comprises:

[0242] The semantic relevance scores between the question-and-answer query terms and the web page titles of the search results are calculated according to the semantic matching model, and the search results are sorted according to the semantic relevance scores.

[0243] A26. The method as described in A25, wherein the method further comprises:

[0244] If the calculated semantic relevance scores are all lower than the second preset value, the output of the labeled candidate answers corresponding to the question-and-answer query terms according to the machine reading comprehension model and subsequent steps will not be performed, and the selected summary of the question-and-answer query terms will be directly captured from the question-and-answer site.

[0245] A27. The method as described in A25, wherein the method further comprises the following steps of training the semantic matching model:

[0246] Obtain question-answer query terms and web page title pairs with positive and negative examples, and construct them into training data through the processor dictionary;

[0247] Based on the BERT pre-trained model and the training data, fine-tune adjustment training is performed to obtain the semantic matching model.

[0248] A28. The method as described in A1, wherein the method further comprises:

[0249] For question-and-answer query words whose PV is lower than the third preset value, selected snippets are directly captured from question-and-answer sites.

[0250] A29. The method as described in A1, wherein the method further comprises:

[0251] Generalizing the question-answer query words to obtain query words with similar semantics;

[0252] The marked answer is also used as a selected summary of a query word with similar semantics in a search engine.

[0253] A30. The method as described in A29, wherein the generalization of the question-answer query words to obtain semantically similar query words comprises:

[0254] Based on the query word display situation, user click behavior and total click behavior, mining candidate query words corresponding to the question-and-answer query words;

[0255] The semantic relevance scores of the question-and-answer query term and each of the candidate query terms are calculated according to the semantic matching model, and the candidate query terms with semantic relevance scores higher than a fourth preset value are regarded as semantically similar query terms.

[0256] A31. The method as described in A29, wherein the generalization of the question-answer query words to obtain semantically similar query words comprises:

[0257] The query words are represented by vectors, and the candidate query words corresponding to the question-answering query words are determined by calculating the cosine similarity of each vector;

[0258] The semantic relevance scores of the question-and-answer query term and each of the candidate query terms are calculated according to the semantic matching model. If the highest semantic relevance score is greater than a fifth preset value, the candidate query term corresponding to the highest semantic relevance score is used as a query term with similar semantics.

[0259] A32. The method as described in A1, wherein the step of obtaining the labeled answer corresponding to the labeled candidate answer based on active learning comprises:

[0260] Provides precise mark interface and rough mark interface;

[0261] Displaying a unique candidate answer for annotation through the rough annotation interface, receiving returned correctness evaluation information, and determining the annotation answer according to the correctness evaluation information;

[0262] as well as,

[0263] Multiple candidate annotation answers are displayed through the precise annotation interface, and the returned annotation answers are received.

[0264] A33. The method as described in any one of A1 to A32, wherein the step of using the marked answer as a selected summary of the corresponding question-answer query in the search engine comprises:

[0265] Saves the featured snippet in xml format.

[0266] The embodiment of the present invention further discloses B34, a device for generating a search engine selected summary, comprising:

[0267] an identification unit, adapted to identify question-answering query terms from search log data;

[0268] A search unit, adapted to obtain search results corresponding to the question-and-answer query term;

[0269] a candidate unit adapted to, if the search result does not contain a document of a specified type, take the question-and-answer query and the corresponding search result as input data, and output a labeled candidate answer corresponding to the question-and-answer query according to a machine reading comprehension model;

[0270] The selected summary generating unit is adapted to obtain the annotated answer corresponding to the annotated candidate answer based on active learning, and use the annotated answer as the selected summary of the corresponding question-and-answer query in the search engine.

[0271] B35. A device as described in B34, wherein

[0272] The identification unit is suitable for performing preset type of processing on the search log data to extract query words that meet the requirements; taking the query words as input data, and outputting question-answer query words according to the query word classification model.

[0273] B36. The device as described in B35, wherein:

[0274] The identification unit is adapted to sort the query words in the search log data according to the page views PV, and extract several query words from high to low in the sorting; and / or normalize the query words to remove punctuation and / or spaces in the query words.

[0275] B37. The device as described in B35, wherein the device further comprises:

[0276] The training unit is suitable for obtaining the search log data generated within a preset time period as sample data; counting the total number of clicks of each query word in the sample data and the number of clicks on question-and-answer sites, and calculating the question-and-answer click ratio of each query word in the sample data within the preset time period; taking the query word whose question-and-answer click ratio is greater than a first threshold as a positive example, and taking the query word whose question-and-answer click ratio is lower than a second threshold as a negative example, to obtain training data.

[0277] B38. The device as described in B37, wherein:

[0278] The training unit is suitable for dividing the training data into a training set, a validation set and a test set according to a preset ratio, and performing training based on the textCNN model to obtain the query term classification model.

[0279] B39. The device as described in B37, wherein:

[0280] The training unit is adapted to take the query words containing the designated words in the sample data as positive examples.

[0281] B40. The device as described in B37, wherein the question-and-answer site is determined based on a URL pattern.

[0282] B41. A device as described in B34, wherein

[0283] The recognition unit is adapted to pre-process the recognized question-answer query words, remove duplicate question-answer query words, and remove question-answer query words for which marked answers have been obtained.

[0284] B42. A device as described in B34, wherein

[0285] The recognition unit is adapted to classify the recognized question-answer query words and filter out question-answer query words other than preset types.

[0286] B43. The device as described in B42, wherein:

[0287] The recognition unit is adapted to classify the recognized question-and-answer query terms according to at least one of a topic classification model, a query term classification model, and an answer classification model.

[0288] B44. A device as described in B43, wherein the subject type includes at least one of the following: mobile digital, life, games, education science, leisure and hobbies, culture and art, financial management, social livelihood, sports, and regions; the query term types include: fact type and / or opinion type; the answer type includes: description type and / or entity type.

[0289] B45. A device as described in B44, wherein

[0290] The candidate unit is adapted to call a machine reading comprehension model that does not include a sorting algorithm to output multiple labeled candidate answers if the answer type of the question-and-answer query term is an entity class; otherwise, call a complete machine reading comprehension model to output one labeled candidate answer.

[0291] B46. A device as described in B43, wherein the topic classification model is a text multi-classification model, and the device also includes: a training unit, adapted to obtain search log data generated within a preset time period as sample data; counting the click ratio of each query word in the sample data on sites of different topic types; counting the page views of each query word in the sample data, and dividing high-frequency query words from the query words in the sample data according to the page views; and taking the topic type of the site with the highest click ratio of the high-frequency query word as the topic type of the corresponding high-frequency query word.

[0292] B47. The device as described in B46, wherein

[0293] The training unit is suitable for dividing medium-frequency query words from the query words of the sample data according to the page views; using the high-frequency query words with determined subject types as training sets, training a support vector machine SVM model, and determining the subject types of the medium-frequency query words according to the trained SVM model.

[0294] B48. The device as described in B46, wherein

[0295] The training unit is adapted to classify low-frequency query words from the query words of the sample data according to the page views; and determine the subject type of the low-frequency query words according to a syntactic dependency tree.

[0296] B49. An apparatus as described in B43, wherein the query term classification model and the answer classification model are both text multi-classification models; the query term classification model is trained based on the nature characteristics of the query terms; and the answer classification model is trained based on the length characteristics of the answer.

[0297] B50. The device as described in B34, wherein:

[0298] The recognition unit is adapted to filter the recognized question-answer query words according to a number of specified dimensions to obtain filtered question-answer query words.

[0299] B51. A device as described in B50, wherein:

[0300] The recognition unit is suitable for filtering each designated dimension using a corresponding text classification model; the text classification model is obtained by training based on SVM and / or fastText.

[0301] B52. The device as described in B34, wherein:

[0302] The search unit is suitable for calling a search engine interface to obtain a first number of natural search results corresponding to the question-and-answer query terms according to the order of search results; adjusting the natural search results according to a preset algorithm, and selecting a second number of natural search results therefrom as search results corresponding to the corresponding question-and-answer query terms.

[0303] B53. The device as described in B52, wherein:

[0304] The search unit is adapted to filter out document sites that cannot obtain webpage content from the natural search results; and to improve the order of sites with a trustworthiness higher than a first preset value.

[0305] B54. A device as described in B52, wherein

[0306] The search unit is adapted to filter the question-and-answer query terms according to the natural search results.

[0307] B55. A device as described in B54, wherein

[0308] The search unit is suitable for filtering out question-and-answer query words containing application boxes in natural search results; and / or filtering out question-and-answer query words containing illegal words in the titles of natural search results; and / or, based on semantic matching, filtering out question-and-answer query words that lack high-quality natural search results.

[0309] B56. The device as described in B34, wherein:

[0310] The candidate unit is further adapted to directly extract the marked candidate answers from the documents of the specified type if the search results include the documents of the specified type.

[0311] B57. An apparatus as described in B56, wherein the document of the specified type is an HTML document containing several pieces of step description information, and the candidate unit is suitable for parsing the HTML document, extracting the several pieces of step description information based on field matching, and obtaining the labeled candidate answers by splicing.

[0312] B58. The device as described in B34, wherein

[0313] The candidate unit is adapted to calculate the semantic relevance score between the question-and-answer query term and the web page title of the search result according to a semantic matching model, and sort the search results according to the semantic relevance score.

[0314] B59. The device as described in B58, wherein

[0315] The candidate unit is adapted to not execute the steps of outputting the labeled candidate answer corresponding to the question-answer query term according to the machine reading comprehension model and subsequent steps if the calculated semantic relevance scores are all lower than a second preset value;

[0316] The selected summary unit is suitable for directly fetching the selected summary of the question-and-answer query term from the question-and-answer site.

[0317] B60. The device as described in B58, wherein the device further comprises:

[0318] The training unit is suitable for obtaining question-answer query terms and web page title pairs containing positive and negative examples annotated, and constructing them into training data through a processor dictionary; based on the BERT pre-trained model and the training data, fine-tune adjustment training is performed to obtain the semantic matching model.

[0319] B61. A device as described in B34, wherein

[0320] The selected summary unit is suitable for directly fetching selected summaries from question-and-answer websites for question-and-answer query words whose PV is lower than a third preset value.

[0321] B62. A device as described in B34, wherein

[0322] The selected summary unit is also suitable for generalizing the question-answer type query words to obtain query words with similar semantics; and using the marked answers as selected summaries of the query words with similar semantics in the search engine.

[0323] B63. A device as described in B62, wherein

[0324] The featured summary unit is suitable for mining candidate query terms corresponding to the question-and-answer query terms based on query term display, user click behavior and co-click behavior; calculating the semantic relevance score between the question-and-answer query terms and each of the candidate query terms according to a semantic matching model, and taking the candidate query terms with a semantic relevance score higher than a fourth preset value as semantically similar query terms.

[0325] B64. A device as described in B62, wherein

[0326] The featured summary unit is suitable for representing the query terms as vectors, and determining the candidate query terms corresponding to the question-and-answer query terms by calculating the cosine similarity of each vector; calculating the semantic relevance scores of the question-and-answer query terms and each of the candidate query terms according to the semantic matching model, and if the highest semantic relevance score is greater than a fifth preset value, the candidate query term corresponding to the highest semantic relevance score is used as a query term with similar semantics.

[0327] B65. The device as described in B34, wherein:

[0328] The selected summary unit is suitable for providing a fine mark interface and a rough mark interface; displaying a unique marking candidate answer through the rough mark interface, receiving returned correctness evaluation information, and determining the marking answer based on the correctness evaluation information; and displaying multiple marking candidate answers through the fine mark interface and receiving the returned marking answers.

[0329] B66. A device as described in any one of B34-B65, wherein

[0330] The selected summary unit is suitable for storing the selected summary in XML format.

[0331] An embodiment of the present invention also discloses C67, an electronic device, wherein the electronic device comprises: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to execute a method as described in any one of A1-A33.

[0332] An embodiment of the present invention also discloses D68, a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, implement the method as described in any one of A1-A33.

Claims

1. A method for generating a search engine featured summary, comprising: Identify question-answering query terms from search log data; Obtain search results corresponding to the question-and-answer query term; If the search results do not contain a document of a specified type, the question-answer query and the corresponding search results are used as input data, and a labeled candidate answer corresponding to the question-answer query is output according to a machine reading comprehension model; the document of the specified type is an HTML document containing a number of step description information; Acquire an annotated answer corresponding to the annotated candidate answer based on active learning, and use the annotated answer as a selected summary of the corresponding question-answer query in the search engine; The step of obtaining the annotated answer corresponding to the annotated candidate answer based on active learning includes: Provides precise mark interface and rough mark interface; Displaying a unique candidate answer for annotation through the rough annotation interface, receiving returned correctness evaluation information, and determining the annotation answer according to the correctness evaluation information; and, Multiple candidate annotation answers are displayed through the precise annotation interface, and the returned annotation answers are received.

2. The method of claim 1, wherein: The step of identifying question-answering query words from the search log data includes: Perform preset processing on search log data to extract query words that meet the requirements; The query words are used as input data, and question-answering query words are output according to the query word classification model.

3. The method of claim 2, wherein: The processing of the search log data by a preset type to extract the query words that meet the requirements includes: Sort the query words in the search log data by page views PV, and extract several query words from high to low in the order; and / or, Normalize the query words and remove punctuation and / or spaces in the query words.

4. The method of claim 2, wherein: The method also includes the following steps of preparing training data for the query term classification model: Obtaining search log data generated within a preset time period as sample data; Counting the total number of clicks on each query word in the sample data and the number of clicks on question-and-answer sites, and calculating the percentage of question-and-answer clicks on each query word in the sample data within a preset time period; The query words whose question-answer click ratio is greater than the first threshold are taken as positive examples, and the query words whose question-answer click ratio is lower than the second threshold are taken as negative examples to obtain training data.

5. The method of claim 4, wherein: The method also includes the following steps of training to obtain the query term classification model: The training data is divided into a training set, a validation set and a test set according to a preset ratio, and is trained based on a textCNN model to obtain the query term classification model.

6. The method of claim 4, wherein: The step of preparing the training data of the query word classification model also includes: The query words containing the designated words in the sample data are taken as positive examples.

7. The method of claim 4, wherein: The question-and-answer site is determined according to a URL pattern.

8. The method of claim 1, wherein: The step of identifying question-answer query words from the search log data further includes: The identified question-and-answer query words are preprocessed to remove duplicate question-and-answer query words and question-and-answer query words that have been annotated with answers.

9. The method of claim 1, wherein: The step of identifying question-answer query words from the search log data further includes: The identified question-and-answer query words are classified and the question-and-answer query words other than the preset types are filtered out.

10. The method of claim 9, wherein: The classifying the identified question-answer query words includes: The identified question-and-answer query terms are classified according to at least one of a topic classification model, a query term classification model, and an answer classification model.

11. The method of claim 10, wherein: The subject types include at least one of the following: mobile digital, life, games, education science, leisure and hobbies, culture and art, financial management, social livelihood, sports, and regions; The query word types include: factual and / or opinion; Answer types include: description class and / or entity class.

12. The method of claim 11, wherein: The step of taking the question-answer query and the corresponding search results as input data and outputting the labeled candidate answers corresponding to the question-answer query according to the machine reading comprehension model comprises: If the answer type of the question-and-answer query term is an entity class, a machine reading comprehension model that does not include a sorting algorithm is called to output multiple labeled candidate answers; otherwise, a complete machine reading comprehension model is called to output one labeled candidate answer.

13. The method of claim 10, wherein: The topic classification model is a text multi-classification model, and the method further includes the following steps of preparing training data for the topic classification model: Obtaining search log data generated within a preset time period as sample data; Counting the click ratio of each query word in the sample data on sites of different subject types; Counting the page views of each query word in the sample data, and classifying high-frequency query words from the query words in the sample data according to the page views; The subject type of the site with the highest click ratio for the high-frequency query word is used as the subject type of the corresponding high-frequency query word.

14. The method of claim 13, wherein: The step of preparing the training data of the topic classification model also includes: Dividing the query words of the sample data into medium-frequency query words according to the page views; The high-frequency query words with determined subject types are used as training sets to train a support vector machine (SVM) model, and the subject types of the medium-frequency query words are determined based on the trained SVM model.

15. The method of claim 13, wherein: The step of preparing the training data of the topic classification model also includes: Classify low-frequency query words from the query words in the sample data according to the page views; The topic type of the low-frequency query word is determined according to the syntactic dependency tree.

16. The method of claim 10, wherein: The query term classification model and the answer classification model are both text multi-classification models; The query word classification model is obtained by training based on the property characteristics of the query words; The answer classification model is trained based on the length feature of the answer.

17. The method of claim 1, wherein: The step of identifying question-answer query words from the search log data further includes: The identified question-and-answer query words are filtered according to a number of specified dimensions to obtain filtered question-and-answer query words.

18. The method of claim 17, wherein: The filtering of the identified question-answer query words according to several specified dimensions includes: For each designated dimension, a corresponding text classification model is used for filtering; the text classification model is obtained by training based on SVM and / or fastText.

19. The method of claim 1, wherein: The obtaining of search results corresponding to the question-answer query term includes: Calling a search engine interface to obtain a first number of natural search results corresponding to the question-and-answer query term according to the search result order; The natural search results are adjusted according to a preset algorithm, and a second number of natural search results are selected therefrom as search results corresponding to the corresponding question-and-answer query terms.

20. The method of claim 19, wherein: The adjusting the natural search results according to a preset algorithm includes at least one of the following: Filter out document sites that cannot obtain web page content from the natural search results; The order of sites whose trustworthiness is higher than the first preset value is promoted.

21. The method of claim 19, wherein: The step of obtaining search results corresponding to the question-and-answer query term further includes: The question-and-answer query terms are filtered according to the natural search results.

22. The method of claim 21, wherein: The filtering of the question-and-answer query words according to the natural search results includes at least one of the following: Filter out Q&A search terms that contain app boxes in natural search results; Filter out Q&A search terms that contain illegal words in the titles of natural search results; Based on semantic matching, filter out question-and-answer search terms that lack high-quality natural search results.

23. The method of claim 1, wherein: The method further includes: If the search results include documents of a specified type, the labeled candidate answers are directly extracted from the documents of the specified type.

24. The method of claim 23, wherein: The step of directly extracting the annotation candidate answer from the document of the specified type includes: The HTML document is parsed, the plurality of step description information are extracted according to field matching, and the labeled candidate answers are obtained by splicing.

25. The method of claim 1, wherein: The using the question-answer query words and the corresponding search results as input data includes: The semantic relevance scores between the question-and-answer query terms and the web page titles of the search results are calculated according to the semantic matching model, and the search results are sorted according to the semantic relevance scores.

26. The method of claim 25, wherein: The method further comprises: If the calculated semantic relevance scores are all lower than the second preset value, the output of the labeled candidate answers corresponding to the question-and-answer query terms according to the machine reading comprehension model and subsequent steps will not be performed, and the selected summary of the question-and-answer query terms will be directly captured from the question-and-answer site.

27. The method of claim 25, wherein: The method further comprises the following steps of training the semantic matching model: Obtain question-answer query terms and web page title pairs with positive and negative examples, and construct them into training data through the processor dictionary; Based on the BERT pre-trained model and the training data, fine-tune adjustment training is performed to obtain the semantic matching model.

28. The method of claim 1, wherein: The method further includes: For question-and-answer query words whose PV is lower than the third preset value, selected snippets are directly captured from question-and-answer sites.

29. The method of claim 1, wherein: The method further includes: Generalizing the question-answer query words to obtain query words with similar semantics; The marked answer is also used as a selected summary of a query word with similar semantics in a search engine.

30. The method of claim 29, wherein: The query words with similar semantics obtained by generalizing the question-answer query words include: Based on the query word display situation, user click behavior and total click behavior, mining candidate query words corresponding to the question-and-answer query words; The semantic relevance scores of the question-and-answer query term and each of the candidate query terms are calculated according to the semantic matching model, and the candidate query terms with semantic relevance scores higher than a fourth preset value are regarded as semantically similar query terms.

31. The method of claim 29, wherein: The query words with similar semantics obtained by generalizing the question-answer query words include: The query words are represented by vectors, and the candidate query words corresponding to the question-answering query words are determined by calculating the cosine similarity of each vector; The semantic relevance scores of the question-and-answer query term and each of the candidate query terms are calculated according to the semantic matching model. If the highest semantic relevance score is greater than a fifth preset value, the candidate query term corresponding to the highest semantic relevance score is used as a query term with similar semantics.

32. The method of any one of claims 1 to 31, wherein: The method of using the marked answer as a selected summary of the corresponding question-answer query in the search engine includes: Saves the featured snippet in xml format.

33. A device for generating a search engine selected summary, comprising: an identification unit, adapted to identify question-answering query terms from search log data; A search unit, adapted to obtain search results corresponding to the question-and-answer query term; A candidate unit, adapted to, if the search results do not contain a document of a specified type, take the question-answer query and the corresponding search results as input data, and output a labeled candidate answer corresponding to the question-answer query according to a machine reading comprehension model; the document of the specified type is an HTML document containing a plurality of step description information; A selected summary generating unit is adapted to obtain an annotated answer corresponding to the annotated candidate answer based on active learning, and use the annotated answer as a selected summary of the corresponding question-answer query in the search engine; wherein, The selected summary unit is suitable for providing a fine mark interface and a rough mark interface; displaying a unique marking candidate answer through the rough mark interface, receiving returned correctness evaluation information, and determining the marking answer based on the correctness evaluation information; and displaying multiple marking candidate answers through the fine mark interface and receiving the returned marking answers.

34. The apparatus of claim 33, wherein: The identification unit is suitable for performing preset type of processing on the search log data to extract query words that meet the requirements; taking the query words as input data, and outputting question-answer type query words according to the query word classification model.

35. The apparatus of claim 34, wherein: The identification unit is adapted to sort the query words in the search log data according to the page views PV, and extract several query words from high to low in the sorting; and / or normalize the query words to remove punctuation and / or spaces in the query words.

36. The apparatus of claim 34, wherein: The device also includes: The training unit is suitable for obtaining the search log data generated within a preset time period as sample data; counting the total number of clicks of each query word in the sample data and the number of clicks on question-and-answer sites, and calculating the question-and-answer click ratio of each query word in the sample data within the preset time period; taking the query word whose question-and-answer click ratio is greater than a first threshold as a positive example, and taking the query word whose question-and-answer click ratio is lower than a second threshold as a negative example, to obtain training data.

37. The apparatus of claim 36, wherein: The training unit is suitable for dividing the training data into a training set, a validation set and a test set according to a preset ratio, and performing training based on the textCNN model to obtain the query term classification model.

38. The apparatus of claim 36, wherein: The training unit is adapted to take the query words containing the designated words in the sample data as positive examples.

39. The apparatus of claim 36, wherein: The question-and-answer site is determined according to a URL pattern.

40. The apparatus of claim 33, wherein: The recognition unit is adapted to pre-process the recognized question-answer query words, remove duplicate question-answer query words, and remove question-answer query words for which marked answers have been obtained.

41. The apparatus of claim 33, wherein: The recognition unit is adapted to classify the recognized question-answer query words and filter out the question-answer query words other than the preset types.

42. The apparatus of claim 41, wherein: The recognition unit is adapted to classify the recognized question-and-answer query terms according to at least one of a topic classification model, a query term classification model, and an answer classification model.

43. The apparatus of claim 42, wherein: The topic types include at least one of the following: mobile digital, life, games, education science, leisure and hobbies, culture and art, financial management, social livelihood, sports, and regions; the query word types include: facts and / or opinions; the answer types include: descriptions and / or entities.

44. The apparatus of claim 43, wherein: The candidate unit is adapted to call a machine reading comprehension model that does not include a sorting algorithm to output multiple labeled candidate answers if the answer type of the question-and-answer query term is an entity class; otherwise, call a complete machine reading comprehension model to output a labeled candidate answer.

45. The apparatus of claim 42, wherein: The topic classification model is a text multi-classification model, and the device also includes: a training unit, which is suitable for obtaining search log data generated within a preset time period as sample data; counting the click ratio of each query word in the sample data on sites of different topic types; counting the page views of each query word in the sample data, and dividing high-frequency query words from the query words in the sample data according to the page views; and taking the topic type of the site with the highest click ratio of the high-frequency query word as the topic type of the corresponding high-frequency query word.

46. ​​The apparatus of claim 45, wherein: The training unit is suitable for dividing medium-frequency query words from the query words of the sample data according to the page views; using the high-frequency query words with determined subject types as training sets, training a support vector machine SVM model, and determining the subject types of the medium-frequency query words according to the trained SVM model.

47. The apparatus of claim 45, wherein: The training unit is adapted to classify low-frequency query words from the query words of the sample data according to the page views; and determine the subject type of the low-frequency query words according to a syntactic dependency tree.

48. The apparatus of claim 42, wherein: The query term classification model and the answer classification model are both text multi-classification models; the query term classification model is trained based on the nature characteristics of the query terms; and the answer classification model is trained based on the length characteristics of the answers.

49. The apparatus of claim 33, wherein: The recognition unit is adapted to filter the recognized question-answer query words according to a number of specified dimensions to obtain filtered question-answer query words.

50. The apparatus of claim 49, wherein: The recognition unit is suitable for filtering each designated dimension using a corresponding text classification model; the text classification model is obtained by training based on SVM and / or fastText.

51. The apparatus of claim 33, wherein: The search unit is adapted to call a search engine interface to obtain a first number of natural search results corresponding to the question-and-answer query term according to a search result sequence; The natural search results are adjusted according to a preset algorithm, and a second number of natural search results are selected therefrom as search results corresponding to the corresponding question-and-answer query terms.

52. The apparatus of claim 51, wherein: The search unit is adapted to filter out document sites that cannot obtain webpage content from the natural search results; and to improve the order of sites with a trustworthiness higher than a first preset value.

53. The apparatus of claim 51, wherein: The search unit is adapted to filter the question-and-answer query terms according to the natural search results.

54. The apparatus of claim 53, wherein: The search unit is suitable for filtering out question-and-answer query words containing application boxes in natural search results; and / or filtering out question-and-answer query words containing illegal words in the titles of natural search results; and / or, based on semantic matching, filtering out question-and-answer query words that lack high-quality natural search results.

55. The apparatus of claim 33, wherein: The candidate unit is further adapted to directly extract the marked candidate answers from the documents of the specified type if the search results include the documents of the specified type.

56. The apparatus of claim 55, wherein: The candidate unit is suitable for parsing the HTML document, extracting the plurality of step description information according to field matching, and obtaining the labeled candidate answers by splicing.

57. The apparatus of claim 33, wherein: The candidate unit is adapted to calculate the semantic relevance score between the question-answer query term and the web page title of the search result according to the semantic matching model, and sort the search results according to the semantic relevance score.

58. The apparatus of claim 57, wherein: The candidate unit is adapted to not execute the steps of outputting the labeled candidate answer corresponding to the question-answer query term according to the machine reading comprehension model and subsequent steps if the calculated semantic relevance scores are all lower than a second preset value; The selected summary unit is suitable for directly fetching the selected summary of the question-and-answer query term from the question-and-answer site.

59. The apparatus of claim 57, wherein: The device also includes: The training unit is suitable for obtaining question-answer query terms and web page title pairs containing positive and negative examples annotated, and constructing them into training data through a processor dictionary; based on the BERT pre-trained model and the training data, fine-tune adjustment training is performed to obtain the semantic matching model.

60. The apparatus of claim 33, wherein: The selected summary unit is suitable for directly fetching selected summaries from question-and-answer websites for question-and-answer query words whose PV is lower than a third preset value.

61. The apparatus of claim 33, wherein: The selected summary unit is also suitable for generalizing the question-answer type query words to obtain query words with similar semantics; and using the marked answers as selected summaries of the query words with similar semantics in the search engine.

62. The apparatus of claim 61, wherein: The featured summary unit is suitable for mining candidate query terms corresponding to the question-and-answer query terms based on query term display, user click behavior and co-click behavior; calculating the semantic relevance score between the question-and-answer query terms and each of the candidate query terms according to a semantic matching model, and taking the candidate query terms with a semantic relevance score higher than a fourth preset value as semantically similar query terms.

63. The apparatus of claim 61, wherein: The featured summary unit is suitable for representing the query terms as vectors, and determining the candidate query terms corresponding to the question-and-answer query terms by calculating the cosine similarity of each vector; calculating the semantic relevance scores of the question-and-answer query terms and each of the candidate query terms according to the semantic matching model, and if the highest semantic relevance score is greater than a fifth preset value, the candidate query term corresponding to the highest semantic relevance score is used as a query term with similar semantics.

64. The device of any one of claims 33 to 63, wherein: The selected summary unit is suitable for storing the selected summary in XML format.

65. An electronic device, wherein: The electronic device comprises: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor performs the method as claimed in any one of claims 1 to 32.

66. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, which, when executed by a processor, implement the method according to any one of claims 1 to 32.

Images