Antithetical Learning for Question Answering (QA)

Abstract

The present disclosure relates to antithesis learning for question answering (QA), and proposes a method and apparatus for providing antithesis training data. A positive example can be obtained from a training dataset, the positive example comprising a first text and a second text marked as related. Antithesis information can be extracted from a search log. The first text can be modified based on at least the antithesis information. The modified first text and the second text can be combined into a counterexample that is antithetical to the positive example, wherein the modified first text and the second text are marked as unrelated in the counterexample.

Description

Background Art

[0001] Search engines can provide search results for user queries in search results pages (SERPs). Traditional search results include links to web documents that are most relevant to the user's query. Web documents may also be referred to as web pages, for example. Links may refer to hyperlinks, web addresses, URLs, etc. To find answers relevant to their queries, users must view search results, click on links to web documents, and browse the presented web documents. In recent years, some web search engines have begun offering question answering (QA) services, also known as web QA services. QA services provide a more efficient information access mechanism by extracting the most relevant passages from web documents and presenting the passage content directly to users. For example, if a user's query contains a question, the web search engine will extract the most relevant passages from the web document and place them in a separate QA block in the SERP. A passage may refer to one or more sentences, one or more paragraphs, or a summary extracted from the corresponding web document. QA services are becoming increasingly popular among search engine users and are becoming an important service provided by search engines. Summary of the Invention

[0002] This summary is provided to introduce a set of concepts that will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Embodiments of the present disclosure provide a method and apparatus for providing parallelism training data. A positive example can be obtained from a training dataset, the positive example comprising a first text and a second text marked as related. Parallelism information can be extracted from a search log. The first text can be modified based at least on the parallelism information. The modified first text and the second text can be combined into a counterexample that parallels the positive example, wherein the modified first text and the second text are marked as unrelated in the counterexample.

[0004] It should be noted that one or more of the above aspects include features described in detail below and particularly pointed out in the claims. The following description and drawings set forth in detail certain illustrative features of the one or more aspects. These features are merely indicative of the various ways in which the principles of the various aspects may be implemented, and the present disclosure is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The disclosed aspects will be described below with reference to the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects.

[0006] Figure 1 An exemplary search results page is shown.

[0007] Figure 2 An exemplary process of providing battle training data according to an embodiment is shown.

[0008] Figure 3 An exemplary process of providing confrontation training data through a web knowledge-based method (WKM) according to an embodiment is shown.

[0009] Figure 4 An exemplary process of candidate item extraction according to an embodiment is shown.

[0010] Figure 5 Exemplary semi-structured data according to an embodiment is shown.

[0011] Figure 6 An exemplary process of providing battle training data through a user feedback based method (UFM) according to an embodiment is shown.

[0012] Figure 7 A flowchart of an exemplary method for providing battle training data according to an embodiment is shown.

[0013] Figure 8 An exemplary apparatus for providing battle training data according to an embodiment is shown.

[0014] Figure 9 An exemplary apparatus for providing battle training data according to an embodiment is shown. DETAILED DESCRIPTION

[0015] The present disclosure will now be discussed with reference to various exemplary embodiments. It should be understood that the discussion of these embodiments is only for enabling those skilled in the art to better understand and thereby implement the embodiments of the present disclosure, and does not teach any limitation on the scope of the present disclosure.

[0016] Typically, a QA system can be configured within a search engine to provide Web QA services, thereby providing the most relevant passages to a query in the SERP. The QA system can include a QA model, also known as a QA relevance model. The QA model is used to assign a relevance score to each candidate passage, based on the relevance score of each candidate passage. The QA system can then select the passage that is most relevant to the query based on the relevance score of each candidate passage, and present it to the user within a QA block in the SERP.

[0017] Web QA can be understood as a text matching task, and text matching can broadly refer to technologies used to identify whether a pair of texts are semantically related. Some traditional approaches use information retrieval (IR) models as QA models to provide QA services. IR models may include, for example, vector space models, BM 25 models, language models for IR, and the like. In other approaches, in order to cope with the diversity of user query expressions, neural network models are used as QA models. Neural network models can also refer to deep learning models, deep neural network models, and the like. The neural network model encodes the semantic meaning of the query text into a vector. By mapping similar expressions to adjacent positions in the vector space, the neural network model can more accurately recall text segments related to the query. In addition, deep pre-training methods can be used to further improve the performance of the neural network model through context embedding.

[0018] Neural network models capture semantic similarity between texts based on distributional assumptions. For example, they assume that language units with similar distributions have similar meanings. Thus, while a neural network model may successfully learn that "kid" is similar to "child," it may also assume that "elderly" is similar to both "kid" and "children," because the set of words that frequently co-occur with "elderly" is also likely to appear in the context of both "kid" and "children." Using word2vec for word embedding, for example, the closest words to the word "adult" in the vector space might include "youth," "child," and "elderly." This shows that word embedding techniques not only cluster synonyms but also tend to cluster other words in the same category in the vector space. Even with deep contextual embedding, as used in deep pre-training, a neural network model may still consider "child" and "elderly" to be similar, as the contexts of these two words often overlap.

[0019] Therefore, the neural network model may not have sufficient sensitivity to distinguish between words that are associated with attributes or categories but have different meanings. For example, the words "children" and "elderly" are both associated with the category "people" or the attribute "people", but their meanings are different. In the scenario of web QA, the lack of such sensitivity may lead to a poor user experience. For example, if the user query is "cold treatment for children", the search engine may provide a paragraph about how to treat children's colds, which is relevant to the query, that is, a good answer to the query. However, when the user query is "cold treatment for the elderly", since the neural network model cannot effectively distinguish between "elderly" and "children", the search engine may still provide a paragraph about how to treat children's colds, which will be irrelevant to the query, that is, an inappropriate answer to the query.

[0020] Adversarial training is commonly used for neural network models. Adversarial training aims to ensure that small perturbations that do not change the meaning of the input text will not significantly alter the model's output. For example, adversarial examples can be generated by modifying words in the original text. However, adversarial training only enhances the model's robustness and cannot be used to improve its sensitivity.

[0021] The embodiments of the present disclosure propose to enhance the sensitivity of the QA model, such as the sensitivity of the neural network model for web QA tasks, through contrastive learning, so that the model can have the ability to effectively distinguish words that are related in attributes or categories but have different meanings. In this article, contrastive learning may include, for example, automatically constructing or providing contrastive training data for enhancing the sensitivity of the model, and training the model using the automatically constructed contrastive training data. In addition, in this article, "contrast" is used to describe the relationship between two texts. For example, two contrasting texts may refer to two texts that are related in attributes or categories but are not similar in meaning. For example, "children" and "elderly" are two contrasting words.

[0022] The training dataset for training a QA model may include training data in the form of <q, p, label>, where q represents a query or question, p represents a passage, and "label" indicates the relevance between q and p. When q and p are labeled as relevant, the <q, p> pair can be used as a positive example, and when q and p are labeled as irrelevant, the <q, p> pair can be used as a negative example. Given a positive example <q1, p1> in the training dataset, where q1 and p1 are labeled as relevant, embodiments of the present disclosure can automatically generate a query q1' that is对仗 to q1, and construct an anti-example <q1', p1> that is对仗 to the positive example <q1, p1>, where q1' and p1 are labeled as irrelevant. In this example, since q1' is对仗 to q1, q1' has a semantic shift from q1. The constructed anti-example can be added to the training dataset for training the QA model. As a specific example, assume that q1 is "the treatment of children's colds" and p1 is a passage on how to treat children's colds. Embodiments of the present disclosure can construct q1' as, for example, "the treatment of the elderly's colds", and accordingly form an anti-example consisting of the query "the treatment of the elderly's colds" and the passage on how to treat children's colds. By generating anti-examples that are对仗 to the positive examples in the training dataset, the QA model can learn both what information should be associated from the positive examples and what information should be distinguished from the constructed对仗 anti-examples, for example, distinguishing words that are associated in terms of attributes or categories but not similar in meaning.

[0023] Embodiments of the present disclosure can mine对仗 information from search logs and use the对仗 information to construct anti-examples that are对仗 to the positive examples in the training dataset. Two unsupervised methods for automatically constructing对仗 training data are proposed, for example, the web knowledge-based method (WKM) and the user feedback-based method (UFM). WKM can use the set of对仗 word pairs mined from search logs to generate对仗 training data by means of word replacement. For example, WKM can obtain candidate items from search logs, at least use the semi-structured data corpus collected on the web to cluster the candidate items into multiple groups, and form对仗 word pairs from the candidate items included in each group. WKM can use the mined set of对仗 word pairs to modify the query in the positive example, such as replacing the words in the query, to form an anti-example. UFM can at least select a对仗 query that is对仗 to the query in the positive example according to search records in the search logs, such as the displayed links, the user's click behavior, etc., and use the selected对仗 query to form an anti-example.

[0024] It should be noted that the term "对仗" in the original text seems to be a specific concept in the relevant field, but its exact meaning is not clear. Here, it is directly translated as "对仗" while maintaining the integrity of the text. If there is a more accurate English equivalent, it may need to be adjusted according to the specific context.It should be understood that the embodiments of the present disclosure are not limited to constructing parallel <query, paragraph> pairs in QA scenarios, but can be more widely used to construct parallel text pairs <text 1', text 2> for various types of text pairs <text 1, text 2> in other application scenarios. For example, a search engine may need to determine whether two queries have the same meaning, or any other model may need to determine whether two texts have the same meaning. By constructing training text pairs as counterexamples that are opposite to the training text pairs as positive examples according to the embodiments of the present disclosure, the model's sensitivity in determining the similarity of text meanings can be enhanced. In addition, for example, when a search engine needs to classify queries or any other model needs to classify text, by utilizing the parallel training data constructed according to the embodiments of the present disclosure, it can help improve the model's ability to identify changes in meaning caused by word changes in the text, thereby enhancing the model's sensitivity. Therefore, although the construction of parallel query-paragraph pair training data is used as an example in some of the sections discussed below, the same or similar processing procedures can also be applied to the construction of any other text pair training data for parallelism.

[0025] Figure 1 An exemplary search results page (SERP) 100 is shown. This SERP 100 may be presented to a user by a search engine in a user interface in response to a user query or question. The components within SERP 100 may be exemplarily divided into a search block 110, a Q&A block 120, a related question block 130, a web link block 140, and so on. Here, a block is merely a logical division of the components within SERP 100. In terms of display and functionality, the different blocks and the components therein may be independent of each other or combined with each other.

[0026] In search block 110 , a user may enter a query, such as “summer flu treatment.”

[0027] In response to determining that the user input in the search block 110 has a question intent, the search engine may provide a QA block 120 in the SERP 100. The QA block 120 may include, for example, a text segment 122 for answering the user query, expansion options 124 for the text segment 122, a source page link 126 for the text segment 122, etc. The text segment 122 is the content most relevant to the user query extracted from the web document. For example, in Figure 1In the example, a text segment 122 may include multiple tips for treating summer colds. Due to page display size limitations, text segment 122 may only be partially displayed. In this case, the user can click an expansion option 124, such as a "More Items" link, to view the hidden portion of text segment 122. A source page link 126 is a hyperlink to the source page or web document from which text segment 122 was extracted. When a user clicks source page link 126, the source page of text segment 122 may be presented in the user interface. Optionally, SERP 100 may also include a feedback button or link 128 to collect user satisfaction feedback regarding text segment 122.

[0028] The related questions block 130 may include questions that are related or similar to the user query in the search block 110. These related questions may include, for example, queries that are frequently searched by other users. Figure 1 In the example, multiple questions related to the user query "summer flu treatment" are shown in the related question block 130, such as "What causes summer flu?", "Medications for summer flu?", etc. When the user clicks on a related question, the search engine may initiate a search for the clicked related question and present the corresponding SERP in the user interface.

[0029] The web link block 140 includes hyperlinks to web pages or web documents related to the user query in the search block 110. The web links in the web link block 140 can be ranked by the search engine based on document relevance. When the user clicks on a web link, the web page can be presented in the user interface.

[0030] It should be understood that Figure 1 All blocks and components in the SERP 100 are exemplary. Depending on specific design and application requirements, the SERP 100 may include more or fewer blocks and components, and the blocks and components may be arranged and presented in any other manner.

[0031] Figure 2 An exemplary process 200 for providing parallel training data according to an embodiment is shown. The process 200 can be executed to automatically generate parallel counterexamples for text pairs in a training dataset 210 that are positive examples, so as to expand the training data in the training dataset 210.

[0032] Training dataset 210 may include training data for training a target model. The target model may be any model related to text relevance prediction, such as a QA model in a QA system, a text meaning comparison model, or a text classification model. The training data in training dataset 210 may take the form of <Text 1, Text 2, Tag>, where "Tag" indicates the relevance between Text 1 and Text 2. When the tag is "relevant," the text pair may be used as a positive example, while when the tag is "irrelevant," the text pair may be used as a negative example.

[0033] In the process 200 , it is assumed that an exemplary positive example 212 “<text 1, text 2, related>” is taken from the training data set 210 , where text 1 and text 2 are marked as related, and it is assumed that a counterexample corresponding to the positive example 212 is desired to be generated.

[0034] In actual applications, there may be a large number of interactions between users and search engines, and information related to these interactions may be stored in the search log 220. The search log 220 may include a search record for each query. A search record may include various information in the SERP provided in response to the query, such as the query, the text provided in response to the query, and the webpage links provided in response to the query. The search record may also include various user behaviors on the SERP, such as clicks on webpage links.

[0035] At 230, paraphrase information 240 may be extracted from the search log 220. As used herein, paraphrase information may refer to various information useful for generating paraphrase training data. In one embodiment, paraphrase information 240 may include a set of paraphrase word pairs, which may be used in a WKM to generate paraphrase training data. In another embodiment, paraphrase information 240 may include a paraphrase query, which may be used in a UFM to generate paraphrase training data.

[0036] At 250, the text 1 in the positive example 212 can be modified based on at least the antithesis information 240. The text 1 can be modified in different ways. For example, a word in the text 1 can be replaced with a word in the antithetical word pair set. For example, the antithetical query can be used to directly replace the text 1.

[0037] Modified text 1 can be used to form a counterexample 252 that counters positive example 212. For example, modified text 1 can be combined with text 2 in positive example 212 to form a text pair <modified text 1, text 2>. Since modified text 1 counters text 1, modified text 1 and text 2 can be marked as unrelated. Thus, counterexample 252 "<modified text 1, text 2, unrelated>" is formed. Counterexample 252 can serve as counterexample training data for positive example 212.

[0038] By repeatedly executing the process 200 , an adversarial training data set including a plurality of adversarial training data can be automatically obtained.

[0039] Process 200 can then add the adversarial training data in the adversarial training data set to the training data set 210. The adversarial training data set can be added to the training data set 210 in different ways. In one embodiment, the adversarial training data set can be simply appended to the training data set 210 as additional training data in addition to the original training data in the training data set 210. In one embodiment, the training data set 210 can be updated by using at least a portion of the adversarial training data in the adversarial training data set to replace a portion of the original negative examples in the training data set 210, so as to guide sensitivity training and ensure the original accuracy of the model while also ensuring a balance between the number of positive examples and the number of negative examples in the training data set. The negative examples in the updated training data set can be configured according to the following formula:

[0040]

[0041] Among them, X n is the original set of counterexamples in the training dataset, X * is the set of antithesis training data generated by process 200, is the final set of counterexamples in the updated training dataset, sample(L,K) is the sampling function used to sample K instances from the source L, |X n | represents X n , α is the sampling coefficient, which is used to control the ratio of selecting examples from the original counterexamples and from the adversarial training data. For example, when α=0.2, it means that 20% of the original counterexamples in the training dataset are replaced by the adversarial training data generated by process 200.

[0042] It should be understood that all operations, steps, and their order in process 200 are exemplary, and process 200 can be modified in any manner according to specific application requirements and designs. For example, although process 200 only shows the generation of a counterexample 252 for positive example 212, it is also possible to perform multiple modifications on text 1 at 250 using different parallelism information and obtain different versions of the modified text 1, thereby generating multiple counterexamples for positive example 212. In addition, depending on the task to be performed by the trained model, the text pair <text1, text2> targeted by process 200 can be any text pair involving text relevance prediction, such as the <query, paragraph> pair in a QA scenario, the <query, query> pair in a search engine query meaning comparison scenario, the <sentence, sentence> pair or the <word, word> pair in a general language model text meaning comparison scenario, and so on.

[0043] Figure 3 FIG3 shows an exemplary process 300 for providing battle training data by WKM according to an embodiment. Figure 2 An exemplary implementation of process 200 is shown. For example, Figure 3 The process of mining antithetical word pairs from search logs can be regarded as Figure 2 A specific implementation of the process of extracting antithesis information from search logs.

[0044] The search log 310 may include a record of a user's search session. A search session may refer to a search process established between a search engine and a user for one or more interrelated queries. A search session involving more than one query may be referred to as a multi-round search session. In a multi-round search session, the search engine may first receive an initial query input by the user, i.e., a first-round query. After being presented with the SERP for the first-round query, the user may wish to modify the first-round query to obtain further information, thereby initiating a second-round query associated with the first-round query. After obtaining the second-round query, the search engine may perform a search and return the SERP for the second-round query. Similarly, the user may continue to provide further queries. Since a series of queries in a multi-round search session will be good candidates for generating antithetical word pairs, process 300 may generate an antithetical word pair set based on the multi-round search sessions in the search log.

[0045] At 320, candidate extraction can be performed in the search log 310. For example, at least one multi-round search session can be first extracted from the search log. In one embodiment, the at least one multi-round search session can have the same first-round query. Candidates can be extracted from the extracted at least one multi-round search session. Candidates can refer to candidates that can serve as words in a set of antithetical word pairs. It should be understood that in this document, "words" can broadly refer to characters, words, phrases, etc. As an example, candidate 1, candidate 2, candidate 3, ... and so on can be extracted from the search log 310 at 320. Figure 4 An exemplary process 400 of candidate extraction according to an embodiment is shown.

[0046] Figure 4 The search log 410 in may correspond to Figure 3 310 in the search log 410. Assume that multiple multi-round search sessions, for example, Session 1, Session 2, and Session 3, are extracted from the search log 410, each including the same first-round search for "diabetes." Session 1 includes multi-round queries, for example, first-round query 422 for "diabetes," second-round query 424 for "type 1 diabetes," and third-round query 426 for "diabetes symptoms." Session 2 includes multi-round queries, for example, first-round query 432 for "diabetes," second-round query 434 for "diabetes treatment," and third-round query 436 for "diabetes treatment for women." Session 3 includes multi-round queries, for example, first-round query 442 for "diabetes," second-round query 444 for "type 1 diabetes symptoms," and third-round query 446 for "type 1 diabetes fatigue."

[0047] In process 400, for each multi-round search session, the shared terms in each two adjacent queries can be extracted as the body, and the unshared terms in each two adjacent queries can be extracted as candidates. In one embodiment, the longest common sequence (LCS) can be used to detect the shared terms in two adjacent queries. Taking session 1 as an example, query 422 "diabetes" and query 424 "type 1 diabetes" share the LCS "diabetes". An entry B1 corresponding to the LCS "diabetes" can be established in the subject set B, and other terms in the two queries, such as "type 1", can be stored as candidates in the subset O1 of the candidate set O corresponding to B1. Similarly, the query 424 "type 1 diabetes" and the query 426 "diabetes symptoms" share the LCS "diabetes". The subject set B already has an entry B1 corresponding to the LCS "diabetes". Therefore, other terms in the two queries, such as "type 1" and "symptoms", can be stored as candidates in the subset O1 corresponding to B1. Since the candidate "type 1" already exists in the subset O1, the repeated storage of "type 1" can be avoided. By performing the above subject and candidate extraction for each multi-round search session, the following can be obtained: Figure 4 The table at the bottom shows the subject information and candidate information, for example, the candidate item set O1 corresponding to the subject item B1 "diabetes" is {type 1, symptoms, treatment, female}, the candidate item set O2 corresponding to the subject item B2 "type 1 diabetes" is {symptoms, fatigue}, the candidate item set O3 corresponding to the subject item B3 "diabetes treatment" is {female}, etc. The information in the table can be indexed and further represented as { i ,O i,j >|i=1,2,…,M;j=1,2,…,N}, where B i represents the i-th subject, M is the number of subjects, O i,j Represents the candidate set O corresponding to the i-th subject i The jth candidate in N is the set of candidates corresponding to the i-th subject O i The number of candidates in .

[0048] Return to Figure 3 , the candidates extracted at 320 may include, for example Figure 4 The candidate items in the candidate item sets O1, O2, and O3.

[0049] ​In some cases, the candidate items extracted at 320 may not be suitable for directly forming antithetical word pairs. In one aspect, some candidate items may not belong to the same category or attribute and cannot be used to form antithetical word pairs. For example, "type 1" and "female" do not belong to the same category or attribute, and it would be meaningless to generate a <type 1, female> word pair, which is not an antithetical word pair. In another aspect, some candidate items may be synonyms and should not be used to form antithetical word pairs. For example, "woman" and "female" are synonyms and have similar meanings, and thus, the formation of <woman, female> word pairs should be avoided. Taking the above into account, process 300 may also include data optimization of the extracted candidate items.

[0050] At 330, group clustering may be performed on the candidate items. For example, the candidate items extracted at 320 may be clustered into group 1, group 2, ..., etc. Each group may include one or more candidate items having the same category or attribute.

[0051] In one embodiment, group clustering can be performed using a pre-prepared semi-structured data corpus 332. The semi-structured data corpus 332 may include different types of semi-structured data obtained from the web, such as web tables, web lists, web menus, etc. Typically, candidates belonging to the same category or attribute will co-occur in the same semi-structured data. Figure 5 Exemplary semi-structured data according to an embodiment is shown. A web table 512 is displayed on a web page 510. As highlighted by a dotted line, web table 512 includes words such as "Phase 1," "Phase 2," "Phase 3," and "Phase 4" belonging to the same category or attribute. Furthermore, a web list 522 is displayed on a web page 520. As highlighted by a dotted line, web list 522 also includes words such as "Phase 1," "Phase 2," "Phase 3," and "Phase 4" belonging to the same category or attribute.

[0052] The similarity between the two candidate items can be calculated based on at least the occurrence information of the two candidate items in the semi-structured data corpus 332. For example, given a candidate item pair ( i ,o j ) and semi-structured data d∈D, where o i and o j are the two candidate items extracted at 320, and D represents the semi-structured data corpus. Based on the inclusion relationship between the candidate items and the semi-structured data, that is, the occurrence of the candidate items in the semi-structured data, the following definition can be given:

[0053]

[0054] Where X∈{1,0} indicates dl Does it contain o i , and |D| is the amount of semi-structured data in corpus D.

[0055] Furthermore, the following two distributions can be defined:

[0056]

[0057] Among them, X∈{1,0} indicates whether d∈D contains o i , and Y∈{1,0} indicates whether d∈D contains o j .

[0058] Based on the above distribution, the mutual information of two candidates (o i ,o j ) similarity score between:

[0059]

[0060] Given a set of candidate items O, we can generate groups and determine the group corresponding to each candidate item by greedy clustering. We can first select o1∈O as a group. Then, we traverse o i ∈O(i≠1), and calculate o i With each existing group G j ∈C, the similarity score is calculated as follows:

[0061]

[0062] Among them, G j is an existing group, |G j |It is Group G j The number of candidates in It is Group G j The nth candidate in . If the maximum similarity score S(o i ,G j )<τ, where τ is a predetermined threshold derived from the empirical average of a set of seed adversarial word pairs, then o i As a new group, otherwise, o i Add to the group G with the maximum similarity score j .

[0063] Table 1 below shows an exemplary process of determining the group corresponding to each candidate item through greedy clustering.

[0064]

[0065] Table 1

[0066] Through the clustering at 330, we can obtain the group set C = {G1, G2, ..., G |C|}, where |C| is the number of groups. For example, the groups in group set C may correspond to Figure 3 Group 1, Group 2, and so on.

[0067] Optionally, at 340, intra-group deduplication can be performed on the groups generated at 330 to remove synonymous candidates. A group including two or more synonymous candidates can be first identified, and then only one of the two or more synonymous candidates is retained in the group. For example, if a group includes the synonyms "woman" and "female", one of the two words, such as "woman", can be removed, and only the other word "female" can be retained in the group. In one embodiment, WordNet can be used to remove synonyms from a group. After the processing at 340, a group set C′={G1′, G2′, …, G |C′| ′}, which can correspond to Figure 3 Group 1', Group 2' and so on.

[0068] At 350, any two candidates in each group can be combined into an antithetical word pair. In the case of performing intra-group deduplication at 340, multiple antithetical word pairs 352-1 can be generated based on the candidates in group 1', multiple antithetical word pairs 352-2 can be generated based on the candidates in group 2', and so on. All the antithetical word pairs obtained can form an antithetical word pair set 352. For example, each group G can be k Any two candidates in ′∈C′ are paired together to obtain the antithetical word pair set PairSet={ <o i ,o j >|i,j∈G k It should be understood that, without performing intra-group deduplication at 340 , the antithetical word pair set 352 may be generated directly based on the group 1, group 2, etc. obtained at 330 .

[0069] The antithesis word pair set 352 is an example of antithesis information extracted from the search log. Therefore, steps 320 to 350 in process 300 can be regarded as Figure 2 An exemplary implementation of step 230 is shown.

[0070] After obtaining the antithetical word pair set 352, the antithetical word pair set 352 can be used to modify the positive example 370 from the training data set. For example, a target word in text 1 in the positive example 370 can be identified at 360, and the target word is also included in an antithetical word pair 360-1 in the antithetical word pair set 352. In one embodiment, the unigram words, bigram words, trigram words, etc. in text 1 can be traversed, and a target word in text 1 that matches a word in an antithetical word pair in the antithetical word pair set 352 can be found. The other word in the antithetical word pair can be regarded as the antithetical word of the target word in text 1. At 380, the antithetical word in the antithetical word pair 360-1 can be used to replace the target word in text 1 to achieve the modification of text 1. Accordingly, a counterexample 390 that is opposed to the positive example 370 can be obtained, which includes a text pair consisting of the modified text 1 and text 2 that is marked as irrelevant.

[0071] It should be understood that multiple antithetical word pairs may be determined for a target word at 360, so that multiple counterexamples can be formed using these antithetical word pairs. In addition, there may be multiple target words in Text 1, so that modifications to Text 1 can be performed for each of these target words and multiple counterexamples can be formed accordingly.

[0072] As an example, assuming that the positive example 370 is a text pair consisting of the query "treatment of children's colds" and a text paragraph about how to treat children's colds, through process 300, it may be determined that "children" in the query is the target word and is included in the antithetical word pair such as <children, elderly>. Therefore, the "children" in the query can be replaced with "elderly", and a counterexample consisting of the query "treatment of elderly colds" and a text paragraph about how to treat children's colds is constructed.

[0073] Figure 6 An exemplary process 600 for providing battle training data via UFM according to an embodiment is shown. Figure 2 An exemplary implementation of process 200 is shown. For example, Figure 6 The process of mining antithetical queries from search logs can be regarded as Figure 2 A specific implementation of the process of extracting antithesis information from search logs.

[0074] Search logs can not only record the search results for user queries, but also record the user's behavior when interacting with the SERP, such as clicking on web links. By using click information, two different queries can be associated. and query For example, where B represents a term shared by two queries, o1 and o2 represent other terms in the two queries, and ° represents the term concatenation. If the two queries share or display some web links in their respective SERPs, but the web links clicked by users differ significantly, it is very likely that terms o1 and o2 are antithetical to each other, and thus, q1 and q2 are also antithetical to each other. For example, suppose q1 is "How many calories are in cola?" The SERP for q1 includes web links {URL1, URL2, URL3}, and the clicked web link is URL1. Meanwhile, suppose q2 is "How many nutrients are in cola?" The SERP for q2 includes web links {URL2, URL3, URL5}, and the clicked web link is URL5. These two queries, respectively, inquire about "calories" and "nutrition" in cola, and therefore have different meanings. The SERPs for these two queries share web links URL2 and URL3, but users click on different web links, URL1 and URL5, respectively. This indicates that the intent of the two queries is different. Therefore, these two queries are likely antithetical to each other.

[0075] The process 600 can be performed to generate a counterexample corresponding to the positive example 610 in the training dataset. At 620, at least one relevant query 624 related to the text 1 in the positive example 610 can be determined from the search log 622. The at least one relevant query 624 determined can be collectively represented as a relevant query set Q r In the QA scenario, text 1 and text 2 in the positive example 610 may be the query and the paragraph, respectively. In one embodiment, a query-based inverted index may be constructed for fast query retrieval in the search log, and BM25 may be used for ranking.

[0076] At 630, the search records corresponding to the positive example 610 and the search records corresponding to the Q r The search records corresponding to each relevant query in the search results may include, for example, queries, text segments, web page links, clicks on web page links, etc.

[0077] At 640, the search record corresponding to the positive example 610 and the search record corresponding to the Q r The search records corresponding to each relevant query in the positive example 610 are calculated by comparing the text 1 with Q r For example, based on the search records, the number of commonly displayed links and the number of commonly clicked links between Text 1 and each related query can be determined, and the value of the comparability parameter between Text 1 and the related query can be calculated based on at least the number of commonly displayed links and the number of commonly clicked links.

[0078] The text 1 can be represented as q, and q and Q can be calculated by the following formula r Related queries in q r Commonly displayed link information, commonly clicked link information, etc.

[0079] CoDisplay(q,q r )=U(q)∩U(q r ) Formula (7)

[0080] UnionDisplay(q,q r )=U(q)∪U(q r ) Formula (8)

[0081] CoClick(q,q r )=Click(q)∩Click(q r ) Formula (9)

[0082] Among them, q r ∈Q r , U(q) represents the list of links provided in the SERP for q, U(q r ) indicates that in response to q r The list of links provided in the SERP for q, Click(q) represents the list of links clicked in the SERP for q, Click(q r ) indicates that in response to q r List of links clicked in SERP, CoDisplay(q,q r ) indicates that for q and q r List of links displayed together in SERP, UnionDisplay(q,q r ) indicates that for q and q r List of all displayed links in SERP, CoClick(q,q r ) indicates that for q and q r A list of commonly clicked links in the SERPs.

[0083] q and q r The more links are shared, the more q and q r However, if the links clicked together are very different, then q and q r Therefore, the following normalization coefficient can be defined:

[0084]

[0085] The normalization coefficient I defined above c (q,q r ) and Ir (q,q r ) can be used as an example of a counterpoint parameter. Accordingly, the value of the normalization coefficient calculated according to the above formula can be used as the value of q and q r The contrast parameter values between them.

[0086] At 650, a value may be determined from Q based on a comparison between the calculated counterpoint parameter value and a predetermined criterion. r Determine at least one antithesis query from Q r In one embodiment, the final set of parallel query can be represented as: And you can select the one that will be added to the The query in:

[0087]

[0088] where I(·) is a sign function that indicates whether q should be r Add to the corresponding q For example, I(·)=1 indicates that q r Add to In the example, I(·)=0 indicates that q should not be r Add to τ1 and τ2 are predetermined thresholds. For example, τ1 can be set to 0 and τ2 can be set to 0.4. The above formula (12) can be used as an example of a predetermined standard. It should be understood that the embodiments of the present disclosure can also adopt any other form of predetermined standard.

[0089] The antithesis query is an example of antithesis information extracted from the search log. Therefore, steps 620 to 650 in process 600 can be regarded as Figure 2 An exemplary implementation of step 230 is shown.

[0090] After obtaining the antithesis query, the antithesis query can be used to modify the positive example 610. For example, at 660, the antithesis query can be used to directly replace Text 1 in the positive example 610. Accordingly, a counterexample 670 can be obtained that is antithetical to the positive example 610 and includes a text pair marked as irrelevant, consisting of the antithesis query and Text 2. It should be understood that if multiple antithesis queries are determined at 650, multiple counterexamples can be generated using each of these antithesis queries.

[0091] As an example, assuming that the positive example 610 is a text pair consisting of the query "treatment of children's colds" and a text paragraph about how to treat children's colds, then through process 600, an antithetical query such as "how to treat colds for the elderly" can be determined, and a counterexample consisting of the query "how to treat colds for the elderly" and a text paragraph about how to treat children's colds can be constructed.

[0092] Figure 7 A flow chart of an exemplary method 700 for providing battle training data is shown, according to an embodiment.

[0093] At 710 , a positive example may be obtained from a training dataset, the positive example including a first text and a second text marked as being related.

[0094] At 720 , antithesis information can be extracted from the search log.

[0095] At 730 , the first text may be modified based at least on the antithesis information.

[0096] At 740 , the modified first text and the second text may be combined into a counterexample to the positive example, wherein the modified first text and the second text are marked as irrelevant in the counterexample.

[0097] In one embodiment, extracting antithetical word information from the search log may include: extracting at least one multi-round search session from the search log; and generating an antithetical word pair set using queries in the at least one multi-round search session.

[0098] The at least one multi-round search session may have the same first-round query.

[0099] Generating a set of antithetical word pairs may include: extracting candidate items from the queries in the at least one multi-round search session; clustering the candidate items into one or more groups using a semi-structured data corpus; and combining any two candidate items in each group into an antithetical word pair.

[0100] The extracting of candidate items may include: for each multi-round search session, extracting unshared terms in every two adjacent queries as candidate items.

[0101] The clustering may include: for two target candidate items among the candidate items, calculating the similarity between the two target candidate items based at least on occurrence information of the two target candidate items in the semi-structured data corpus.

[0102] The clustering may include determining a group corresponding to each candidate item in the candidate items by a greedy clustering method.

[0103] The semi-structured data in the semi-structured data corpus may be of at least one of the following types: a web table, a web list, and a web menu.

[0104] Method 700 may also include identifying a group comprising two or more synonymous candidates; and retaining only one candidate of the two or more synonymous candidates in the group.

[0105] The modifying the first text may include: identifying a target word included in the first text and in an antithetical word pair included in the antithetical word pair set; and replacing the target word with another word in the antithetical word pair in the first text.

[0106] In one embodiment, extracting antithesis information from the search log may include: determining an antithesis query corresponding to the first text from the search log.

[0107] Determining the antithesis query may include: determining at least one related query related to the first text from the search log; for each related query, calculating an antithesis parameter value between the first text and the related query based at least on a search record corresponding to the positive example and a search record corresponding to the related query in the search log; and selecting a related query having an antithesis parameter value that meets a predetermined standard from the at least one related query as the antithesis query.

[0108] The calculation of the parallelism parameter value may include: determining the number of links displayed together and the number of links clicked together between the first text and the related query based on the search records corresponding to the positive example and the search records corresponding to the related query; and calculating the parallelism parameter value between the first text and the related query based at least on the number of links displayed together and the number of links clicked together.

[0109] The modifying the first text may include: replacing the first text with the antithetical query.

[0110] In one embodiment, the training dataset may be used to train a QA model, wherein the first text corresponds to a query and the second text corresponds to a paragraph.

[0111] It should be understood that method 700 may also include any steps / processes for providing combat training data according to the above-mentioned embodiments of the present disclosure.

[0112] Figure 8 An exemplary apparatus 800 for providing battle training data according to an embodiment is shown.

[0113] The apparatus 800 may include: a positive example obtaining module 810 for obtaining a positive example from a training data set, wherein the positive example includes a first text and a second text marked as related; an antithesis information extraction module 820 for extracting antithesis information from a search log; a text modification module 830 for modifying the first text based at least on the antithesis information; and a counterexample generating module 840 for combining the modified first text and the second text into a counterexample that is antithetical to the positive example, wherein the modified first text and the second text are marked as irrelevant in the counterexample.

[0114] In one embodiment, the antithesis information extraction module 820 may be configured to: extract at least one multi-round search session from the search log; and generate a set of antithetical word pairs using queries from the at least one multi-round search session. Generating the set of antithetical word pairs may include: extracting candidate items from the queries in the at least one multi-round search session; clustering the candidate items into one or more groups using a semi-structured data corpus; and combining any two candidate items in each group into an antithetical word pair.

[0115] In one embodiment, the antithesis information extraction module 820 may be configured to determine an antithesis query corresponding to the first text from the search log.

[0116] In addition, the apparatus 800 may further include any other modules configured to provide any operations for performing combat training data.

[0117] Figure 9 An exemplary apparatus 900 for providing battle training data according to an embodiment is shown.

[0118] The device 900 may include at least one processor 910. The device 900 may also include a memory 920 connected to the processor 910. The memory 920 may store computer-executable instructions, which, when executed, enable the processor 910 to: obtain a positive example from a training data set, the positive example including a first text and a second text marked as relevant; extract antithesis information from a search log; modify the first text based on at least the antithesis information; and combine the modified first text and the second text into a counterexample that is antithetical to the positive example, the modified first text and the second text being marked as irrelevant in the counterexample. In addition, the processor 910 may also be configured to perform any other operations of the method for providing antithesis training data according to the above-mentioned embodiments of the present disclosure.

[0119] The embodiments of the present disclosure may be implemented in a non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions that, when executed, cause one or more processors to perform any operations of the method for providing combat training data according to the above-described embodiments of the present disclosure.

[0120] It should be understood that all operations in the method described above are merely exemplary, and the present disclosure is not limited to any operation in the method or the order of these operations, but should cover all other equivalent transformations under the same or similar concept.

[0121] It should also be understood that all modules in the above-described device can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. In addition, any module in these modules can be further divided into submodules or combined together functionally.

[0122] Processor has been described in conjunction with various devices and methods.These processors can be implemented using electronic hardware, computer software or its arbitrary combination.Whether these processors are implemented as hardware or software will depend on specific application and the overall design constraint imposed on the system.As an example, the processor provided in this disclosure, any part of the processor or any combination of processors can be implemented as microprocessor, microcontroller, digital signal processor (DSP), field programmable gate array (FPGA), programmable logic device (PLD), state machine, gate logic, discrete hardware circuit and other suitable processing components configured for performing the various functions described in this disclosure.The function of the processor provided in this disclosure, any part of the processor or any combination of processors can be implemented as software performed by microprocessor, microcontroller, DSP or other suitable platform.

[0123] Software should be broadly considered to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, processes, functions, etc. Software can reside in a computer-readable medium. A computer-readable medium can include, for example, a memory, which can be, for example, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe), an optical disk, a smart card, a flash memory device, a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a register, or a removable disk. Although the memory is shown as being separate from the processor in various aspects provided in the present disclosure, the memory can also be located inside the processor (e.g., a cache or register).

[0124] The above description is provided to enable any person skilled in the art to implement the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents of the elements of the various aspects described in this disclosure that are known or soon to be known to those skilled in the art are intended to be covered by the claims.

Claims

1. A method for providing battle training data, comprising: Obtain a positive example from a training dataset, the positive example comprising a first text and a second text marked as related; Extracting antithesis information from a search log includes: (1) extracting at least one multi-round search session from the search log; and generating an antithetical word pair set using queries in the at least one multi-round search session; or (2) determining an antithetical query corresponding to the first text from the search log; modifying the first text based at least on the antithesis information; and combining the modified first text and the second text into a counterexample to the positive example, wherein the modified first text and the second text are marked as irrelevant in the counterexample; Among them, determining the antithesis query includes: determining at least one related query related to the first text from the search log; For each relevant query, calculating a comparison parameter value between the first text and the relevant query based on at least the search record corresponding to the positive example and the search record corresponding to the relevant query in the search log; and A related query having a contrast parameter value that meets a predetermined standard is selected from the at least one related query as the contrast query.

2. The method according to claim 1, wherein The at least one multi-round search session has a same first-round query.

3. The method according to claim 1, wherein The generating of the antithetical word pair set comprises: extracting candidates from the query in the at least one multi-round search session; clustering the candidate items into one or more groups using a semi-structured data corpus; and Combine any two candidates in each group into an antithetical word pair.

4. The method according to claim 3, wherein: The extraction candidates include: For each multi-round search session, the unshared terms in every two adjacent queries are extracted as candidates.

5. The method according to claim 3, wherein: The clustering includes: For two target candidate items among the candidate items, similarity between the two target candidate items is calculated based on at least occurrence information of the two target candidate items in the semi-structured data corpus.

6. The method according to claim 3, wherein: The clustering includes: The group corresponding to each candidate item in the candidate items is determined by greedy clustering.

7. The method of claim 3, wherein: The semi-structured data in the semi-structured data corpus belongs to at least one of the following types: a web table, a web list, and a web menu.

8. The method of claim 3, further comprising: Identify groups that include two or more synonymous candidates; as well as Only one of the two or more synonymous candidates is retained in the group.

9. The method of claim 1, wherein: Modifying the first text includes: identifying a target word included in the first text and in an antithetical word pair included in the set of antithetical word pairs; and In the first text, the target word is replaced by another word in the antithetical word pair.

10. The method of claim 1, wherein: Calculation of the antithesis parameter values includes: Determining the number of commonly displayed links and the number of commonly clicked links between the first text and the related query based on the search records corresponding to the positive example and the search records corresponding to the related query; and The comparison parameter value between the first text and the related query is calculated based on at least the number of the commonly displayed links and the number of the commonly clicked links.

11. The method of claim 1, wherein: Modifying the first text includes: The first text is replaced by the antithetical query.

12. The method of claim 1, wherein: The training data set is used to train a question answering (QA) model, the first text corresponds to a query, and the second text corresponds to a text segment.

13. A device for providing battle training data, comprising: A positive example obtaining module, configured to obtain a positive example from a training data set, wherein the positive example includes a first text and a second text marked as being related; The antithesis information extraction module is configured to extract antithesis information from a search log, including: (1) extracting at least one multi-round search session from the search log; and generating an antithetical word pair set using queries in the at least one multi-round search session; or (2) determining an antithetical query corresponding to the first text from the search log; a text modification module, configured to modify the first text based at least on the antithesis information; and a counterexample generation module, combining the modified first text and the second text into a counterexample to the positive example, wherein the modified first text and the second text are marked as irrelevant in the counterexample; Among them, determining the antithesis query includes: determining at least one related query related to the first text from the search log; For each relevant query, calculating a comparison parameter value between the first text and the relevant query based on at least the search record corresponding to the positive example and the search record corresponding to the relevant query in the search log; and A related query having a contrast parameter value that meets a predetermined standard is selected from the at least one related query as the contrast query.

14. The apparatus of claim 13, wherein: The generating of the antithetical word pair set comprises: extracting candidates from the query in the at least one multi-round search session; clustering the candidate items into one or more groups using a semi-structured data corpus; and Combine any two candidates in each group into an antithetical word pair.

15. A device for providing battle training data, comprising: at least one processor; as well as a memory storing computer-executable instructions that, when executed, cause the at least one processor to: A positive example is obtained from a training dataset, the positive example including a first text and a second text that are marked as related. Extracting antithesis information from a search log includes: (1) extracting at least one multi-round search session from the search log; and generating an antithetical word pair set using queries in the at least one multi-round search session; or (2) determining an antithetical query corresponding to the first text from the search log, modifying the first text based at least on the antithesis information, and combining the modified first text and the second text into a counterexample to the positive example, wherein the modified first text and the second text are marked as irrelevant in the counterexample; Among them, determining the antithesis query includes: determining at least one related query related to the first text from the search log; For each relevant query, calculating a comparison parameter value between the first text and the relevant query based on at least the search record corresponding to the positive example and the search record corresponding to the relevant query in the search log; and A related query having a contrast parameter value that meets a predetermined standard is selected from the at least one related query as the contrast query.

Images