Corpus construction method and device, electronic equipment and medium

Abstract

The present disclosure provides a corpus construction method and device, electronic equipment and medium, relates to the technical field of computers, and particularly relates to the technical field of question and answer systems. The implementation scheme is as follows: a first text set is acquired; a plurality of texts in the first text set are clustered according to semantic information of the plurality of texts, so as to obtain a plurality of first text subsets; at least one first keyword of at least one part of the first text subsets is acquired; a synonym keyword set is acquired based on the first keyword; and a text in the first text set that matches the synonym keyword set is acquired as a corpus subset in a corpus.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of question-answering system technology, and specifically to a corpus construction method, device, electronic device, non-transitory computer-readable storage medium, and computer program product. Background Art

[0002] In the field of computer technology, users can query information through question-and-answer systems, which can then return answers based on the user's input. These systems contain a corpus (or language database) containing textual material. The quality of this corpus is crucial to the performance of the system.

[0003] The approaches described in this section are not necessarily approaches that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any approach described in this section is prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, the issues raised in this section should not be considered as having been recognized in any prior art. Summary of the Invention

[0004] The present disclosure provides a corpus construction method, apparatus, electronic device, non-transitory computer-readable storage medium, and computer program product.

[0005] According to one aspect of the present disclosure, a method for constructing a corpus is provided. The method includes: obtaining a first text set, the first text set including multiple texts; clustering the multiple texts in the first text set based on semantic information of the multiple texts to obtain multiple first text subsets; obtaining at least one first keyword from at least a portion of the multiple first text subsets; obtaining a synonymous keyword set based on the first keyword, the synonymous keyword set including multiple synonymous keywords with similar meanings to the first keyword; and obtaining texts in the first text set that match the synonymous keyword set as corpus subsets in the corpus.

[0006] According to another aspect of the present disclosure, a corpus construction device is provided. The device includes: a text set acquisition unit configured to acquire a first text set, the first text set including multiple texts; a clustering unit configured to cluster the multiple texts based on semantic information of the multiple texts in the first text set to obtain multiple first text subsets; a keyword acquisition unit configured to acquire at least one first keyword of at least a portion of the multiple first text subsets; a keyword set acquisition unit configured to acquire a synonymous keyword set based on the first keyword, the synonymous keyword set including multiple synonymous keywords with similar meanings to the first keyword; and a text matching unit configured to acquire texts in the first text set that match the synonymous keyword set as corpus subsets in the corpus.

[0007] According to another aspect of the present disclosure, an electronic device is provided, characterized in that it includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned method.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above method when executed by a processor.

[0010] According to one or more embodiments of the present disclosure, the quality of constructing a corpus can be improved.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.

[0013] Figure 1 A schematic diagram illustrating an example system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0014] Figure 2 A flowchart of a method for constructing a corpus according to an embodiment of the present disclosure is shown;

[0015] Figure 3 A flowchart showing a partial process of the method for constructing a corpus according to an embodiment of the present disclosure;

[0016] Figure 4 Another flow chart of the method for constructing a corpus according to an embodiment of the present disclosure is shown;

[0017] Figure 5 Another flow chart of the method for constructing a corpus according to an embodiment of the present disclosure is shown;

[0018] Figure 6 A structural block diagram of a corpus construction device according to an embodiment of the present disclosure is shown; and

[0019] Figure 7 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0020] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] In this disclosure, unless otherwise specified, the use of terms such as "first" and "second" to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements. Such terms are only used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases, based on the context of the description, they may also refer to different instances.

[0022] The terms used in the descriptions of the various examples described in this disclosure are for the purpose of describing specific examples only and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element may be one or more. In addition, the term "and / or" used in this disclosure encompasses any one and all possible combinations of the listed items.

[0023] The corpus for a question-answering system can include manually selected or set text, such as manually selected standard questions and similar questions with similar meanings. In some related technologies, technical experts can manually annotate standard questions and similar questions. This method is costly, inefficient in corpus construction, and overly reliant on the expertise of technical experts. In some related technologies, corpora generated by computers based on existing samples may also contain a high level of undesirable content, resulting in lower corpus quality.

[0024] Based on this, the present disclosure provides a method for constructing a corpus. This method clusters a text collection, determines a first keyword in each subset of the cluster, and then obtains a set of synonymous keywords based on the keywords. Because the keywords included in the synonymous keyword set are similar in meaning to the corresponding first keyword, by obtaining text from the text collection that matches the synonymous keyword set as a corpus subset in the corpus, the text in the corpus subset can be made more relevant to the first keyword, thereby improving the quality of the corpus.

[0025] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 is a schematic diagram illustrating an example system 100 in which the various methods described herein may be implemented, according to an exemplary embodiment.

[0027] refer to Figure 1 , the system 100 includes a client device 110 , a server 120 , and a network 130 communicatively coupling the client device 110 and the server 120 .

[0028] The client device 110 includes a display 114 and a client application (APP) 112 that can be displayed via the display 114. The client application 112 can be an application that needs to be downloaded and installed before running, or a small program (lite app) that is a lightweight application. In the case where the client application 112 is an application that needs to be downloaded and installed before running, the client application 112 can be pre-installed on the client device 110 and activated. In the case where the client application 112 is a small program, the user 102 can directly run the client application 112 on the client device 110 by searching for the client application 112 in the host application (for example, by the name of the client application 112, etc.) or scanning a graphical code (for example, a barcode, QR code, etc.) of the client application 112, without having to install the client application 112. In some embodiments, the client device 110 can be any type of mobile computer device, including a mobile computer, a mobile phone, a wearable computer device (for example, a head-mounted device such as a smart watch or smart glasses), or other types of mobile devices. In some embodiments, the client device 110 can alternatively be a stationary computer device, such as a desktop computer, a server computer, or other types of stationary computer devices.

[0029] The server 120 is typically a server deployed by an Internet Service Provider (ISP) or an Internet Content Provider (ICP). The server 120 may represent a single server, a cluster of multiple servers, a distributed system, or a cloud server that provides basic cloud services (such as cloud databases, cloud computing, cloud storage, and cloud communications). It will be understood that although Figure 1 1. The server 120 is shown communicating with only one client device 110, but the server 120 may provide background services to multiple client devices simultaneously.

[0030] Examples of network 130 include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), and / or a combination of communication networks such as the Internet. Network 130 can be a wired or wireless network. In some embodiments, data exchanged through network 130 is processed using technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. In addition, encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In some embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0031] For the purpose of the embodiments of this disclosure, Figure 1In the example of , the client application 112 may be an intelligent answering application in a question-and-answer system, which may provide various functions based on intelligent answering, such as obtaining user queries, providing answer information, and so on. Correspondingly, the server 120 may be a server used together with the intelligent answering application in the question-and-answer system. The server 120 may provide online question-and-answer services to the client application 112 running in the client device 110 based on the road network data. Alternatively, the server 120 may provide the road network data to the client device 110, and the client application 112 running in the client device 110 may provide local question-and-answer services.

[0032] Figure 2 FIG. 2 shows a flow chart of a method 200 for constructing a corpus according to an embodiment of the present disclosure. Figure 2 As shown, the method 200 includes:

[0033] Step S210: Acquire a first text set, where the first text set includes a plurality of texts;

[0034] Step S220: clustering the multiple texts in the first text set according to semantic information of the multiple texts to obtain multiple first text subsets;

[0035] Step S230: for at least a portion of the first text subsets among the plurality of first text subsets, obtaining at least one first keyword of the at least a portion of the first text subsets;

[0036] Step S240: Based on the first keyword, a set of synonymous keywords is obtained, where the set of synonymous keywords includes multiple synonymous keywords that are similar in meaning to the first keyword; and

[0037] Step S250: Acquire texts in the first text set that match the synonymous keyword set as a corpus subset in the corpus.

[0038] Since the keywords included in the synonymous keyword set are similar in meaning to the corresponding first keyword, by obtaining the text in the first text set that matches the synonymous keyword set as the corpus subset in the corpus, the text in the corpus subset can be made more relevant to the first keyword, thereby improving the quality of the corpus.

[0039] According to some embodiments, the method 200 may include the following steps:

[0040] Acquire a first text set, where the first text set includes a plurality of texts;

[0041] Clustering the plurality of texts in the first text set according to semantic information of the plurality of texts to obtain a plurality of first text subsets;

[0042] For each first text subset in at least a portion of the first text subsets among the plurality of first text subsets, obtaining at least one first keyword of the first text subset;

[0043] Based on at least one first keyword of at least a portion of the first text subset, obtaining a plurality of synonymous keyword sets, each of the plurality of synonymous keyword sets including a plurality of synonymous keywords having a similar meaning to one of the first keywords; and

[0044] For each synonymous keyword set in the plurality of synonymous keyword sets, texts matching the synonymous keyword set in the first text set are obtained as a corpus subset in the corpus.

[0045] The various steps of method 200 are described in detail below.

[0046] In step S210, a first text set input by a user via a terminal device(s) may be received. The first text set may be a text set obtained by recognizing an audio file using automatic speech recognition (ASR) technology. For example, the terminal device may have a recording function and an automatic speech recognition function.

[0047] According to some embodiments, step S220 may include:

[0048] Extracting a first text subset from the first text set and updating the first text set, wherein semantic similarity between different texts in the first text subset is greater than a first threshold;

[0049] Iteratively perform the following operations until the semantic similarity between the texts in the updated first text set is no greater than the first threshold:

[0050] A next first text subset is extracted from the updated first text set, and the first text set is updated again. In the next first text subset, the semantic similarity between different texts is greater than the first threshold.

[0051] Thus, the texts in the first text set are clustered based on the semantic similarity between each text, and texts with semantic similarity greater than a first threshold can be clustered as texts of the same category into the same first text subset. Texts in different categories can have different semantics. This division method can efficiently cluster a large number of texts with similar meanings into the same first text subset, so that the first keywords subsequently obtained for the first text subset have a higher correlation with the meaning of the key information vocabulary in each text in the first text subset, thereby further improving the quality of the corpus.

[0052] For example, the first text set obtained in step S210 may be set A, which contains a large number of texts, and these large number of texts may be texts for different topics and with different semantics. A first text subset C1 may be first extracted from the first text set A, and the first text set A is updated, and the updated first text set is (A-C1). In the first text subset C1, the semantic similarity between each text is greater than a first threshold. In one example, the distance between text semantic vectors can be used to indicate the semantic similarity between texts, and the first threshold may be a distance threshold, where the higher the semantic similarity, the smaller the distance between text semantic vectors.

[0053] Subsequently, the next first text subset C2 can be extracted from the updated first text set (A-C1), and the first text set is updated again. The updated first text set is (A-C1-C2). In the first text subset C2, the semantic similarity between each text is also greater than the above-mentioned first threshold. Then, the next first text subset C3 can be extracted from the updated first text set (A-C1-C2), and the first text set is updated again. The updated first text set is (A-C1-C2-C3). In the first text subset C3, the semantic similarity between each text is also greater than the above-mentioned first threshold.

[0054] Similarly, the above operation can be performed iteratively to extract multiple first text subsets C 4… C n , n is an integer greater than 4. Until the semantic similarity between the texts in the updated first text set is no greater than the above first threshold. n ) can be treated as a separate category because the semantic similarity between them is not high.

[0055] It is understood that the method for clustering the first text set is not limited to the above method. For example, a second text set may be obtained by sampling from the first text set, and the multiple texts in the second text set may be clustered based on the semantic information of the multiple texts in the second text set to obtain multiple first text subsets; and for each first text subset in the multiple first text subsets, at least one first keyword of the first text subset is obtained. The first text set may also be clustered using a clustering algorithm (such as Kmeans).

[0056] According to some embodiments, the method 200 may further include: acquiring the at least a portion of the first text subset from a plurality of first text subsets according to a preset rule.

[0057] For example, according to a preset rule, a plurality of first text subsets (C1, C2, C3, ... C n ) to obtain at least a portion of the first text subset (for example, C1, C2, C3). The preset rule may be, for example, that the semantic similarity between each text is also greater than a preset threshold (the preset threshold is different from the first threshold).

[0058] Thus, at least a portion of the first text subsets is screened out from the multiple first text subsets, and the screened out first text subsets are used to obtain the corresponding first keywords, without considering the remaining first text subsets (the texts in the remaining first text subsets may contain more undesirable semantics and do not meet the user's needs), which can further improve the efficiency of method 200 in constructing the corpus.

[0059] According to some embodiments, the aforementioned preset rule may include: the number of texts included in the first text subset is greater than a second threshold.

[0060] For example, the number of texts included in each of the at least a portion of the first text subsets (e.g., C1, C2, C3) is greater than the second threshold (e.g., 100), while the remaining first text subsets (e.g., C4...C n ) includes each first text subset in which the number of texts is less than or equal to a second threshold (eg, 100).

[0061] Therefore, the first text subsets that include a larger number of texts are screened out from the numerous first text subsets. Since they include a larger number of texts and the semantic similarity between the texts is also higher, the first keywords obtained for this part of the first text subsets (for example, C1, C2, C3) are of higher quality (better reflecting the semantic themes of the texts in the first text set), which can further improve the efficiency and quality of corpus construction.

[0062] Figure 3 FIG. 2 is a flow chart showing a partial process of a method 200 for constructing a corpus according to an embodiment of the present disclosure. Figure 3 As shown, step S230 may include:

[0063] Step S331: Acquire a first standard text from the at least part of the first text subset;

[0064] Step S332: input the first standard text into the semantic similarity model, and obtain at least one first similar text output by the semantic similarity model that is semantically similar to the first standard text; and

[0065] Step S333: Obtain at least one first keyword based on at least one text in the at least part of the first text subset and at least one first similar text.

[0066] The semantic similarity model can be pre-trained, and the semantics of the first similar text generated using the semantic similarity model is close to the semantics of the first standard text, thereby obtaining more semantically similar texts (including the first standard text and the first similar text), and thus obtaining the first keyword based on more semantically similar texts.

[0067] In step S331, the first standard text may be selected by the user or filtered from the first text subset using a model. In one example, the content of the first standard text may be, for example, "How many days of annual leave are there?"

[0068] In step S332, the semantic similarity model outputs at least one first similar text (e.g., “how many days of vacation are there”, “what is the annual vacation like”, “how is the vacation arranged”, etc.) that is semantically similar to the first standard text (“how many days of vacation are there”).

[0069] According to some embodiments, the above step S333 may include: removing preset words from the at least one text and the at least one first similar text to obtain an intermediate text set; and obtaining at least one first keyword based on the intermediate text set.

[0070] The preset words can be, for example, unwanted topic words pre-set by the user, such as greetings and pleasantries like "hello" and "goodbye." After removing the preset words, an intermediate text set can be obtained, and the probability of the intermediate text set containing the desired topic words will be greatly increased.

[0071] According to some embodiments, the step of acquiring at least one first keyword based on the intermediate text set may include: acquiring at least one first keyword based on the word frequency of each word in the intermediate text set.

[0072] For example, at least one first keyword can be obtained from the intermediate text set based on the word frequency of the remaining text. For example, the top three words with the highest word frequency can be extracted as the three first keywords. In one example, the top three words with the highest word frequency can be "holiday," "internship," and "salary."

[0073] According to some embodiments, the above-mentioned step S333 may include: obtaining at least one first keyword from at least one text and at least one first similar text in the at least part of the first text subset by using a term frequency-inverse document frequency (TF-IDF) method.

[0074] The term frequency-inverse document frequency (TF-IDF) method can be used to assess the importance of a word to a text collection. The main idea of ​​TF-IDF is that if a word appears with a high frequency TF in one text and a low frequency in other texts, it is considered to have good category differentiation ability and is suitable for clustering. In one example, the TF-IDF method can be used to extract the three most frequent first keywords (e.g., "holiday," "internship," and "salary") from at least one text in the first text subset and at least one first similar text.

[0075] According to some embodiments, in step S240, based on at least one first keyword in at least a portion of the first text subset, synonyms may be summarized for the at least one first keyword to obtain a corresponding set of synonymous keywords. For example, multiple sets of synonymous keywords M1, M2, and M3 may be obtained. Keyword set M1 may include keywords such as "annual leave" and "holiday"; keyword set M2 may include keywords such as "social security" and "five insurances and one fund"; and keyword set M3 may include keywords such as "internship period" and "probationary period." In some examples, the above-mentioned synonym summary can be performed based on the semantic similarity of the keywords; in some examples, the above-mentioned synonym summary can be performed by the user.

[0076] According to some embodiments, step S250 may include:

[0077] The text including any synonymous keyword in the synonymous keyword set is obtained from the first text set in a regular expression manner, and the obtained text is used as the corpus subset.

[0078] A regular expression, also known as a regular expression, is a text pattern. A regular expression can use a single string to describe and match a series of strings that match a certain syntactic rule, and can be used to retrieve text that conforms to a certain pattern (rule). Thus, for example, the keywords "annual leave" and "holiday" in the keyword set M1 can be used to construct a regular expression string, and then the string can be used to obtain text that includes the keyword "annual leave" or "holiday" from the first text set A. These texts all include one or more desired keywords in the keyword set M1, thereby making the constructed corpus higher in quality.

[0079] Figure 4 FIG. 4 is a flow chart of a method 400 for constructing a corpus according to an embodiment of the present disclosure. Figure 4 As shown, steps S410 to S450 in method 400 are the same as those described above. Figure 2 Steps S210 to S250 in the described method 200 are similar and will not be described again here.

[0080] According to some embodiments, the method 400 may further include:

[0081] Step S460: sampling from the first text set to obtain a second text set;

[0082] Step S470: clustering the multiple texts in the second text set according to the semantic information of the multiple texts in the second text set to obtain multiple second text subsets; and

[0083] Step S480: Obtain at least one second keyword for the multiple second text subsets.

[0084] Furthermore, the above step S440 of acquiring a set of synonymous keywords based on the first keyword may include: acquiring a set of synonymous keywords based on the first keyword and at least one second keyword.

[0085] In the above step S460 , the first text set A may be randomly sampled to obtain a second text set B (ie, a subset of the first text set A).

[0086] In the above step S470, a similar method as that in step S420 can be used to cluster the multiple texts in the second text set; in addition, since the number of texts in the second text set B is less than that in the first text set A, the k-means clustering algorithm can also be used to cluster the multiple texts in the second text set.

[0087] Thus, by sampling from the first text set A to obtain the second text set B, clustering and determining corresponding keywords for both sets A and B, and then obtaining a set of synonymous keywords based on the determined keywords. When clustering set A, a relatively coarse-grained clustering can be performed; when clustering set B, a relatively fine-grained clustering can be performed, thereby better balancing the clustering effect and speed, and further improving the efficiency of corpus construction.

[0088] According to some embodiments, method 400 may include:

[0089] Sampling the first text set to obtain a second text set;

[0090] Clustering the plurality of texts in the second text set according to semantic information of the plurality of texts in the second text set to obtain a plurality of second text subsets; and

[0091] For each second text subset in the plurality of second text subsets, at least one second keyword of the second text subset is obtained.

[0092] Furthermore, the above step S440 of obtaining multiple synonymous keyword sets based on at least one first keyword of the at least part of the first text subset may include: obtaining multiple synonymous keyword sets based on at least one first keyword and at least one second keyword.

[0093] According to some embodiments, step S480 may include:

[0094] Acquire a second standard text from the plurality of second text subsets;

[0095] Inputting the second standard text into the semantic similarity model, and obtaining at least one second similar text output by the semantic similarity model that is semantically similar to the second standard text; and

[0096] At least one second keyword is obtained based on at least one text in the plurality of second text subsets and at least one second similar text.

[0097] The semantic similarity model may be the same as the semantic similarity model used in step S332 described above. Thus, more semantically similar texts (including the second standard text and the second similar text) can be obtained, thereby obtaining the second keyword based on more semantically similar texts.

[0098] It will be understood that the second standard text may be selected by the user or may be screened from the second text subset using a model.

[0099] After constructing a corpus using method 200 or method 400, a standard corpus can be selected from each corpus subset in the corpus as a standard question in the question-answer database. The remaining corpus in the corpus subset can be used as similar corpus similar to the standard corpus, and the similar corpus can be used as similar questions in the question-answer database. The standard corpus and similar corpus can be used to train various models.

[0100] Figure 5 FIG. 5 shows a flow chart of a method 500 for constructing a corpus according to an embodiment of the present disclosure. Figure 5 As shown, the steps S510 to S550 included in the method 500 are the same as those described above. Figure 2 Steps S210 to S250 in the described method 200 are similar and will not be described again here.

[0101] According to some embodiments, the method 500 may further include:

[0102] Step S560: Obtain at least one standard corpus from the corpus subset;

[0103] Step S570: obtaining multiple similar corpora corresponding to at least one standard corpus from the corpus subset by matching; and

[0104] Step S580: Train the model using at least one standard corpus and multiple similar corpora.

[0105] The standard corpus may be one or more. In step S470, based on the similarity between the standard corpus and other corpora in the corpus subset, multiple similar corpora (similar questions) corresponding to each standard corpus (standard question) in the at least one standard corpus (i.e., the similarity meets a preset threshold) can be matched.

[0106] In step S570 , a plurality of similar corpora corresponding to each standard corpus in the at least one standard corpus may be obtained by matching from the corpus subset.

[0107] Furthermore, in step S580, the standard questions and similar questions obtained in steps S560 and S570 can be used as sample data to train a classification model. For example, each standard question can be used as a sample of a category, and each similar question can belong to the same category as the corresponding standard question. Thus, a classification model can be trained to predict the intent of sentences input into the model. This classification model can be used in a knowledge base to provide corresponding answers based on questions posed by users.

[0108] In step S580, the standard questions and similar questions obtained in steps S560 and S570 can also be used as sample data to train an out-of-domain data model. For example, standard questions and similar questions can be labeled as 1, and text in the corpus subset that is not a standard question or similar question can be labeled as 0, thereby constructing a training sample set. The trained out-of-domain data model can predict whether the input text belongs to in-domain data.

[0109] In addition, the standard questions and similar questions obtained in step S560 and step S570 may be used as sample data to train a semantic matching model, which will not be described in detail here.

[0110] According to another aspect of the present disclosure, a device for constructing a corpus is also provided. Figure 6 FIG. 6 shows a structural block diagram of a corpus construction device 600 according to an embodiment of the present disclosure. Figure 6 As shown, the corpus construction device 600 includes:

[0111] The text set acquiring unit 610 is configured to acquire a first text set, where the first text set includes a plurality of texts;

[0112] The clustering unit 620 is configured to cluster the plurality of texts in the first text set according to semantic information of the plurality of texts to obtain a plurality of first text subsets;

[0113] The keyword acquisition unit 630 is configured to acquire at least one first keyword of at least a portion of the first text subsets among the plurality of first text subsets;

[0114] The keyword set acquisition unit 640 is configured to acquire a synonymous keyword set based on the first keyword, where the synonymous keyword set includes a plurality of synonymous keywords having similar meanings to the first keyword; and

[0115] The text matching unit 650 is configured to obtain texts in the first text set that match the synonymous keyword set as a corpus subset in the corpus.

[0116] It should be understood that Figure 6 The various units of the apparatus 600 shown in FIG. 6 can be compared with those in FIG. Figure 2 The steps in the method 200 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 200 are also applicable to the apparatus 600 and the units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0117] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 6 The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program code / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of units 610 to 650 can be implemented together in a system on chip (SoC). SoC can include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and can optionally execute the received program code and / or include embedded firmware to perform functions.

[0118] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned corpus construction method.

[0119] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned corpus construction method.

[0120] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above-mentioned corpus construction method when executed by a processor.

[0121] See also Figure 7 , a structural block diagram of an electronic device 700 that can be used as the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device can be different types of computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0122] Figure 7 FIG. 1 shows a block diagram of an electronic device according to an embodiment of the present disclosure. Figure 7 As shown, the electronic device 700 may include at least one processor 701 , a working memory 702 , an I / O device 704 , a display device 705 , a storage device 706 , and a communication interface 707 , which can communicate with each other via a system bus 703 .

[0123] The processor 701 may be a single processing unit or multiple processing units, all of which may include a single or multiple computing units or multiple cores. The processor 701 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operational instructions. The processor 701 may be configured to retrieve and execute computer-readable instructions stored in the working memory 702, the storage device 706, or other computer-readable media, such as program code of an operating system 702a, program code of an application 702b, and the like.

[0124] The working memory 702 and the storage device 706 are examples of computer-readable storage media for storing instructions that are executed by the processor 701 to implement the various functions described above. The working memory 702 may include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). In addition, the storage device 706 may include a hard drive, a solid-state drive, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network attached storage, storage area networks, etc. The working memory 702 and the storage device 706 may all be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by the processor 701 as a specific machine configured to implement the operations and functions described in the examples herein.

[0125] I / O devices 704 may include input devices and / or output devices. Input devices may be any type of device capable of inputting information to electronic device 700, including but not limited to a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone, and / or remote control. Output devices may be any type of device capable of presenting information, including but not limited to a video / audio output terminal, a vibrator, and / or a printer.

[0126] The communication interface 707 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as Bluetooth. TM devices, 802.11 devices, Wi-Fi devices, WiMAX devices, cellular communication devices, and / or the like.

[0127] The application program 702b in the working register 702 can be loaded to execute the various methods and processes described above. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via the storage device 706 and / or the communication interface 707. When the computer program is loaded and executed by the processor 701, one or more steps of the corpus construction method described above can be performed.

[0128] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable task scheduling device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0130] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0133] A computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0134] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0135] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. In addition, the steps may be performed in an order different from that described in this disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after this disclosure.

Claims

1. A method for constructing a corpus, characterized in that: include: Acquire a first text set, where the first text set includes a plurality of texts; Clustering the plurality of texts in the first text set according to semantic information of the plurality of texts to obtain a plurality of first text subsets; For at least a portion of the plurality of first text subsets, obtaining at least one first keyword of the at least a portion of the first text subsets; Based on the first keyword, obtaining a set of synonymous keywords, wherein the set of synonymous keywords includes a plurality of synonymous keywords having similar meanings to the first keyword; as well as The texts in the first text set that match the synonymous keyword set are obtained as a corpus subset in the corpus.

2. The method according to claim 1, characterized in that Clustering the plurality of texts in the first text set according to semantic information of the plurality of texts to obtain a plurality of first text subsets includes: Extracting a first text subset from the first text set and updating the first text set, wherein semantic similarity between different texts in the first text subset is greater than a first threshold; Iteratively perform the following operations until the semantic similarity between the texts in the updated first text set is no greater than the first threshold: A next first text subset is extracted from the updated first text set, and the first text set is updated again. In the next first text subset, the semantic similarity between different texts is greater than the first threshold.

3. The method according to claim 1 or 2, characterized in that Also includes: The at least a portion of the first text subsets is obtained from the plurality of first text subsets according to a preset rule.

4. The method according to claim 3, characterized in that The preset rules include: The amount of text included in the first text subset is greater than a second threshold.

5. The method according to claim 1, wherein For at least a portion of the plurality of first text subsets, obtaining at least one first keyword of the at least a portion of the first text subsets includes: Acquire a first standard text from the at least a portion of the first text subset; Inputting the first standard text into a semantic similarity model, and obtaining at least one first similar text output by the semantic similarity model that is semantically similar to the first standard text; and The at least one first keyword is obtained based on at least one text in the at least part of the first text subset and the at least one first similar text.

6. The method according to claim 5, characterized in that Acquiring the at least one first keyword based on at least one text in the at least part of the first text subset and the at least one first similar text includes: removing preset vocabulary from the at least one text and the at least one first similar text to obtain an intermediate text set; and At least one first keyword is obtained based on the intermediate text set.

7. The method according to claim 6, characterized in that Based on the intermediate text set, obtaining at least one first keyword includes: The at least one first keyword is obtained based on the word frequency of each word in the intermediate text set.

8. The method according to claim 5, characterized in that Acquiring the at least one first keyword based on at least one text in the at least part of the first text subset and the at least one first similar text includes: The at least one first keyword is obtained from at least one text in the at least part of the first text subset and the at least one first similar text by using a term frequency-inverse document frequency index TF-IDF method.

9. The method according to claim 1, characterized in that Acquiring texts in the first text set that match the synonymous keyword set as a corpus subset in the corpus includes: The text including any synonymous keyword in the synonymous keyword set is obtained from the first text set in a regular expression manner, and the obtained text is used as a corpus subset.

10. The method according to claim 1, characterized in that Also includes: Sampling the first text set to obtain a second text set; Clustering the plurality of texts in the second text set according to semantic information of the plurality of texts in the second text set to obtain a plurality of second text subsets; as well as For the plurality of second text subsets, obtaining at least one second keyword, Wherein, based on the first keyword, obtaining a set of synonymous keywords includes: The synonymous keyword set is obtained based on the first keyword and the at least one second keyword.

11. The method according to claim 10, characterized in that For the plurality of second text subsets, obtaining at least one second keyword includes: Acquire a second standard text from the plurality of second text subsets; Inputting the second standard text into a semantic similarity model, and obtaining at least one second similar text output by the semantic similarity model that is semantically similar to the second standard text; and At least one second keyword is obtained based on at least one text in the plurality of second text subsets and the at least one second similar text.

12. The method according to claim 1, characterized in that Also includes: Acquire at least one standard corpus from the corpus subset; Matching and obtaining a plurality of similar corpora corresponding to the at least one standard corpus from the corpus subset; as well as The model is trained using the at least one standard corpus and the multiple similar corpora.

13. A corpus construction device, characterized in that: include: A text set acquiring unit is configured to acquire a first text set, where the first text set includes a plurality of texts; a clustering unit configured to cluster the plurality of texts in the first text set according to semantic information of the plurality of texts to obtain a plurality of first text subsets; a keyword acquiring unit configured to acquire, for at least a portion of the plurality of first text subsets, at least one first keyword of the at least a portion of the first text subsets; a keyword set acquiring unit configured to acquire a synonymous keyword set based on the first keyword, wherein the synonymous keyword set includes a plurality of synonymous keywords having similar meanings to the first keyword; as well as The text matching unit is configured to obtain texts in the first text set that match the synonymous keyword set as a corpus subset in the corpus.

14. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause a computer to execute the method according to any one of claims 1-12.

16. A computer program product comprising a computer program, characterized in that The computer program implements the method according to any one of claims 1 to 12 when executed by a processor.

Images