Corpus acquisition method, device, electronic device and storage medium

Abstract

The present disclosure proposes a corpus acquisition method, device, electronic device and storage medium. The specific scheme is: receiving a dialogue sentence, and determining a response sentence corresponding to the dialogue sentence; processing the dialogue sentence and the response sentence respectively to obtain a first sentence information corresponding to the dialogue sentence and a second sentence information corresponding to the response sentence; extracting a target corpus corresponding to a preset corpus label from the first sentence information and the second sentence information. Through the present disclosure, the corpus acquisition efficiency and acquisition effect can be effectively improved, and the acquired target corpus can effectively meet the personalized needs of actual business scenarios.

Description

Technical Field

[0001] The present disclosure relates to the technical field of natural language processing, and in particular to a corpus acquisition method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of artificial intelligence technology, various model algorithms in the field of natural language processing (NLP) are becoming more and more advanced. Model training depends on corpus. The higher the quality of the corpus, the larger the quantity, the closer it is to the current real business scenarios, and the closer it is to the real user language expression habits, the better the generalization effect of the trained model in the real business scenarios.

[0003] In the related technology, external purchases are usually relied on to obtain rich corpus.

[0004] In this way, the efficiency of corpus acquisition is low and the cost is high, which cannot meet the personalized needs of real business scenarios. In addition, the source of the corpus cannot be controlled, which may cause the acquired corpus to be incompatible with the real business scenario and the corpus quality to be poor. Summary of the invention

[0005] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, the purpose of the present disclosure is to propose a corpus acquisition method, device, electronic device and storage medium, which can effectively improve the efficiency and effect of corpus acquisition, and enable the acquired corpus to effectively meet the personalized needs of actual business scenarios.

[0007] To achieve the above-mentioned purpose, the corpus acquisition method proposed in the first aspect of the embodiment of the present disclosure includes: receiving a dialogue sentence and determining a response sentence corresponding to the dialogue sentence; processing the dialogue sentence and the response sentence respectively to obtain first sentence information corresponding to the dialogue sentence and second sentence information corresponding to the response sentence; extracting the target corpus corresponding to the preset corpus label from the first sentence information and the second sentence information.

[0008] The corpus acquisition method proposed in the embodiment of the first aspect of the present disclosure receives a dialogue sentence, determines a response sentence corresponding to the dialogue sentence, processes the dialogue sentence and the response sentence respectively to obtain first sentence information corresponding to the dialogue sentence and second sentence information corresponding to the response sentence, and extracts the target corpus corresponding to the preset corpus label from the first sentence information and the second sentence information, which can effectively improve the corpus acquisition efficiency and acquisition effect, and enable the acquired corpus to effectively meet the personalized needs of actual business scenarios.

[0009] To achieve the above-mentioned purpose, the corpus acquisition device proposed in the second aspect of the embodiment of the present disclosure includes: a first receiving module, used to receive a dialogue sentence and determine a response sentence corresponding to the dialogue sentence; a first processing module, used to process the dialogue sentence and the response sentence respectively to obtain first sentence information corresponding to the dialogue sentence and second sentence information corresponding to the response sentence; an extraction module, used to extract the target corpus corresponding to the preset corpus label from the first sentence information and the second sentence information.

[0010] The corpus acquisition device proposed in the second aspect of the embodiment of the present disclosure receives dialogue sentences and determines response sentences corresponding to the dialogue sentences, respectively processes the dialogue sentences and the response sentences to obtain first sentence information corresponding to the dialogue sentences and second sentence information corresponding to the response sentences, and extracts target corpus corresponding to preset corpus labels from the first sentence information and the second sentence information, which can effectively improve the corpus acquisition efficiency and acquisition effect, and enable the acquired corpus to effectively meet the personalized needs of actual business scenarios.

[0011] The third aspect embodiment of the present disclosure proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the corpus acquisition method proposed in the first aspect embodiment of the present disclosure is implemented.

[0012] The fourth aspect embodiment of the present disclosure proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the corpus acquisition method proposed in the first aspect embodiment of the present disclosure is implemented.

[0013] The fifth aspect of the present disclosure provides a computer program product. When an instruction processor in the computer program product executes, the corpus acquisition method provided in the first aspect of the present disclosure is performed.

[0014] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0016] Figure 1 is a flowchart of a corpus acquisition method proposed in an embodiment of the present disclosure;

[0017] Figure 2 is a schematic diagram of the architecture of a corpus acquisition device according to an embodiment of the present disclosure;

[0018] Figure 3 is a flowchart of a corpus acquisition method proposed in another embodiment of the present disclosure;

[0019] Figure 4 is a flowchart of a corpus acquisition method proposed in another embodiment of the present disclosure;

[0020] Figure 5 is a schematic diagram of a corpus acquisition interface according to an embodiment of the present disclosure;

[0021] Figure 6 is a schematic diagram of a corpus acquisition record interface according to an embodiment of the present disclosure;

[0022] Figure 7 is a flowchart of a corpus acquisition method according to an embodiment of the present disclosure;

[0023] Figure 8 is a structural diagram of a corpus acquisition device proposed in an embodiment of the present disclosure;

[0024] Fig. 9 is a structural schematic diagram of a corpus acquisition device proposed in another embodiment of the present disclosure;

[0025] Fig.10 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present disclosure, and are not to be construed as limitations of the present disclosure. On the contrary, the embodiments of the present disclosure include all changes, modifications, and equivalents that fall within the spirit and connotation of the appended claims.

[0027] Figure 1 It is a flowchart of a corpus acquisition method proposed in an embodiment of the present disclosure.

[0028] It should be noted that the executor of the corpus acquisition method of this embodiment is a corpus acquisition device, which can be implemented by software and / or hardware. The device can be configured in an electronic device, which may include but is not limited to a terminal, a server, etc.

[0029] like Figure 1 As shown, the corpus acquisition method includes:

[0030] S101: Receive a dialogue statement and determine a response statement corresponding to the dialogue statement.

[0031] Among them, the user puts forward a consultation statement through the client of the intelligent dialogue system, and the consultation statement can be called a dialogue statement. Correspondingly, the server of the intelligent dialogue system receives the user's consultation statement and can generate a corresponding reply statement based on the user's consultation statement, and the reply statement can be called an answer statement.

[0032] Among them, the intelligent dialogue system refers to a computer system that can realize a coherent dialogue with people. The user can initiate a dialogue and enter consultation statements through the client of the intelligent dialogue system. The server of the intelligent dialogue system can respond to the client's consultation dialogue and make corresponding answers to realize human-computer interaction.

[0033] Optionally, the corpus acquisition method of the embodiment of the present disclosure can be applied to a human-computer interaction system that implements intelligent dialogue (a human-computer interaction system that implements intelligent dialogue, i.e., the above-mentioned intelligent dialogue system). Of course, it can also be applied to any other possible human-computer interaction system without limitation.

[0034] That is to say, the embodiments of the present disclosure support the collection of corpus directly from an online human-computer interaction system that supports artificial intelligence technology, and the online human-computer interaction system that supports artificial intelligence technology usually exists in actual business scenarios, so that the collected target corpus can effectively meet the personalized needs of actual business scenarios.

[0035] After receiving the dialogue statement and determining the response statement corresponding to the dialogue statement, the subsequent steps can be triggered.

[0036] S102: Process the dialogue sentence and the response sentence respectively to obtain first sentence information corresponding to the dialogue sentence and second sentence information corresponding to the response sentence.

[0037] After receiving the dialogue statement and determining the response statement corresponding to the dialogue statement, the dialogue statement and the response statement may be processed separately to obtain first statement information corresponding to the dialogue statement and second statement information corresponding to the response statement.

[0038] Among them, the dialogue sentence will have some associated sentence information, which can be called first sentence information. Correspondingly, the response statement will have some associated sentence information, which can be called second sentence information. The sentence information can be used to describe sentence-related information. The sentence-related information can be, for example, context information, semantics, scene information involved in the sentence, keywords contained in the sentence, and other information.

[0039] For example, the first sentence information or the second sentence information may specifically include: dialogue time information, dialogue content information, etc. involved in the dialogue sentence or the response sentence, without limitation.

[0040] For example, if the content of the dialogue sentence is: "Query Beijing's weather today", and the time corresponding to the dialogue sentence is time t1, then the semantic content of the dialogue sentence and the time t1 corresponding to the dialogue sentence can be used together as the first sentence information, without any restriction.

[0041] Correspondingly, if the content of the response statement is: "Beijing is sunny today", and the time corresponding to the response statement is time t2, the semantic content of the response statement and the time corresponding to the response statement can be used together as the second sentence information without any restriction.

[0042] For example, after receiving a dialogue statement and determining the response statement corresponding to the dialogue statement, the server can parse the first statement information corresponding to the dialogue statement and the second statement information corresponding to the response statement to obtain the first statement information corresponding to the dialogue statement and the second statement information corresponding to the response statement.

[0043] The method of parsing processing may be, for example, a model parsing method, a text analysis method, etc., and there is no limitation to this.

[0044] S103: Extracting a target corpus corresponding to a preset corpus label from the first sentence information and the second sentence information.

[0045] Among them, corpus tags can be used to describe the category of corpus, and preset corpus tags can refer to corpus tags pre-configured by the human-computer interaction system. Preset corpus tags can be, for example: dialogue type tags, dialogue scene tags, dialogue keyword tags. Preset corpus tags can also be adaptively configured according to actual corpus acquisition needs. Therefore, in the embodiment of the present disclosure, since the corpus tags are pre-configured, in the process of collecting corpus from the online human-computer interaction system that supports artificial intelligence technology, the pre-embedded corpus tags can be used to realize the rapid and accurate collection of target corpus, which effectively improves the efficiency of acquiring the target corpus while enabling the collected target corpus to effectively meet the personalized needs of actual business scenarios.

[0046] That is to say, after processing the dialogue sentences and the response sentences respectively to obtain the first sentence information corresponding to the dialogue sentences and the second sentence information corresponding to the response sentences, the corpus corresponding to the preset corpus label can be extracted from the first sentence information and the second sentence information, and the corpus can be called the target corpus.

[0047] For example, if the preset corpus label is a dialogue scene label, the corpus corresponding to the dialogue scene label can be extracted from the first sentence information corresponding to the dialogue sentence and the second sentence information corresponding to the response sentence. The pair of corpora corresponding to the dialogue scene label is the target corpus.

[0048] In this embodiment, by receiving a dialogue statement and determining a response statement corresponding to the dialogue statement, the dialogue statement and the response statement are processed separately to obtain first sentence information corresponding to the dialogue statement and second sentence information corresponding to the response statement, and target corpus corresponding to a preset corpus label is extracted from the first sentence information and the second sentence information, thereby effectively improving the efficiency and effect of corpus acquisition, and enabling the acquired corpus to effectively meet the personalized needs of actual business scenarios.

[0049] Figure 2 is a schematic diagram of the structure of the corpus acquisition device according to an embodiment of the present disclosure. The description of the following embodiments of the present disclosure can be combined with Figure 2 As shown, an online human-computer interaction system supporting artificial intelligence technology is taken as an example, which is an intelligent dialogue system, and the present disclosure does not limit this.

[0050] Among them, after the consulting user initiates a consulting dialogue to the intelligent dialogue system, the target corpus can be extracted from the consulting dialogue according to the corpus acquisition method provided in the embodiment of the present disclosure, and the extracted target corpus can be pushed to the big data platform for storage, so that the corpus acquisition user can acquire the target corpus. Accordingly, the corpus acquisition device may include:

[0051] Artificial Intelligence Model Algorithm Platform: Corpus Acquisition Users can input corpus acquisition requests into the artificial intelligence model algorithm platform, and the artificial intelligence model algorithm platform can convert the user's corpus acquisition request into a corpus acquisition request that is suitable for the corpus acquisition device.

[0052] Query engine Presto, where the query engine Presto is an open source distributed structured query language (SQL) suitable for interactive analytical queries. The query engine Presto is used to connect the corpus acquisition request converted by the artificial intelligence model algorithm platform to the big data processing platform. Then, the big data processing platform can feed back the corpus acquisition results to the artificial intelligence model algorithm platform so that the corpus acquisition users can download them.

[0053] Figure 3 is a flowchart of a corpus acquisition method proposed in another embodiment of the present disclosure, for Figure 3 The description of Figure 2 .

[0054] like Figure 3 As shown, the corpus acquisition method includes:

[0055] S301: Receive a dialogue statement and determine a response statement corresponding to the dialogue statement.

[0056] S302: Process the dialogue sentence and the response sentence respectively to obtain first sentence information corresponding to the dialogue sentence and second sentence information corresponding to the response sentence.

[0057] The description of S301 - S302 can be specifically referred to the above embodiment, which will not be repeated here.

[0058] S303: Determine the dialogue description information according to the first sentence information and the second sentence information.

[0059] After processing the dialogue statement and the response statement separately to obtain the first statement information corresponding to the dialogue statement and the second statement information corresponding to the response statement, the dialogue description information can be determined according to the first statement information and the second statement information.

[0060] Among them, the information used to describe the general nature of the conversation content can be called conversation description information. The above-mentioned sentence information is information related to the sentence, that is, information used to describe the sentence, and the conversation description information can be used to describe the information of the entire conversation. Conversation description information such as conversation scene, conversation semantics, conversation result information, conversation feedback information, etc., is not limited to this.

[0061] For example, if the first sentence information is: "Query the passenger flow of Xinzheng Airport today", and the second sentence information is: "The passenger flow of Xinzheng Airport today is 20,000 people", then the general nature of the overall conversation can be determined as the conversation description information based on the "query, Xinzheng Airport, today, passenger flow" in the first sentence information, and combined with the "today", "passenger flow", and "20,000 people" in the second sentence information (for example, airport passenger flow query, query time "today", the airport is "Xinzheng Airport", and the query result is "20,000 people"), and there is no restriction on this.

[0062] S304: Determine initial corpus tags according to the conversation description information.

[0063] Among them, after obtaining the dialogue description information, the corpus that may be involved in the dialogue description information can be classified in advance based on the classification rules to determine the initial corpus label, that is, the initial corpus label is obtained based on the initial analysis of the dialogue description information, and the above-mentioned preset corpus label is pre-set as a point label in the intelligent dialogue system.

[0064] For example, after obtaining the above conversation description information (for example, airport passenger flow query, query time "today", the airport is "Xinzheng Airport", and the query result is "20,000 people"), it can be determined that the initial corpus tags involve: conversation type tags, conversation scene tags, conversation time tags, and conversation keyword tags, that is, the tags involved in the conversation description information can be determined as the initial corpus tags.

[0065] S305: If the initial corpus label matches the preset corpus label, it is determined that there is a mapping relationship between the dialogue description information and the preset corpus label.

[0066] After determining the initial corpus label according to the conversation description information, the initial corpus label can be compared with the preset corpus label. If the initial corpus label matches the preset corpus label, it can be determined that there is a mapping relationship between the conversation description information and the preset corpus label, so that the content included in the conversation description information can be determined, and it is more likely to contain the target corpus that matches the preset corpus label, thereby effectively assisting in obtaining the target corpus that matches the preset corpus label from the conversation description information, assisting in improving the accuracy of the target corpus, and ensuring the corpus acquisition effect.

[0067] Optionally, in some embodiments, the similarity value between the conversation description information and the label content of the preset corpus label can also be determined. If the similarity value is greater than a set threshold, it can be determined that there is a mapping relationship between the conversation description information and the preset corpus label. Thus, it is possible to accurately and flexibly determine whether the content included in the conversation description information contains the target corpus that matches the preset corpus label, thereby effectively assisting in obtaining the target corpus from the conversation description information.

[0068] Among them, the value used to measure the similarity between the conversation description information and the label content of the preset corpus label can be called a similarity value.

[0069] For example, feature extraction can be performed on the conversation description information and the label content of the preset corpus respectively to obtain a feature vector corresponding to the conversation description information and a feature vector corresponding to the label content of the preset corpus label. By calculating the vector cosine between the above two types of feature vectors, the similarity value between the conversation description information and the label content of the preset corpus label can be obtained, and then the calculated similarity value can be compared with a preset similarity threshold. If the calculated similarity value is greater than the set threshold, it can be determined that there is a mapping relationship between the conversation description information and the preset label corpus.

[0070] In this embodiment, the threshold value may be set accordingly in combination with the real-time business requirements of the corpus acquisition task. The setting method may also be an adaptive setting, which is not limited.

[0071] S306: If there is a mapping relationship between the dialogue description information and the preset corpus label, extract the target corpus corresponding to the preset corpus label from the first sentence information and the second sentence information.

[0072] Among them, if there is a mapping relationship between the dialogue description information and the preset corpus label, it means that the content included in the dialogue description information has a high probability of containing the target corpus matching the preset corpus label. At this time, it can be triggered to extract the target corpus corresponding to the preset corpus label from the dialogue description information, or the target corpus corresponding to the preset corpus label can also be extracted from the first sentence information and the second sentence information, and there is no restriction on this.

[0073] In this way, the timing of extracting the target corpus can be determined based on the actual situation of whether the initial corpus label matches the preset corpus label, so that the extracted target corpus can be more adapted to the preset corpus label. In addition, since the preset corpus label is pre-configured as a embedding label, the required target corpus can be extracted during the application process of the online human-computer interaction system that supports artificial intelligence technology, thereby ensuring the efficiency and accuracy of corpus acquisition and the adaptability of the acquired target corpus to the actual business scenarios.

[0074] In this embodiment, by receiving a dialogue sentence and determining the response sentence corresponding to the dialogue sentence, processing the dialogue sentence and the response sentence respectively to obtain the first sentence information corresponding to the dialogue sentence and the second sentence information corresponding to the response sentence, and extracting the target corpus corresponding to the preset corpus label from the first sentence information and the second sentence information, the efficiency and effect of corpus acquisition can be effectively improved, and the acquired corpus can effectively meet the personalized needs of actual business scenarios. It also realizes determining the timing of extracting the target corpus according to the actual situation of whether the initial corpus label matches the preset corpus label, which can make the extracted target corpus more compatible with the preset corpus label, and because the preset corpus label is pre-configured as a buried point label, the required target corpus can be extracted during the application process of the online human-computer interaction system supporting artificial intelligence technology, thereby ensuring the efficiency and accuracy of corpus acquisition, and ensuring the adaptability of the acquired target corpus to the actual business scenario.

[0075] Figure 4 It is a flowchart of a corpus acquisition method proposed in another embodiment of the present disclosure.

[0076] like Figure 4 As shown, the corpus acquisition method includes:

[0077] S401: Receive a dialogue statement and determine a response statement corresponding to the dialogue statement.

[0078] S402: Process the dialogue sentence and the response sentence respectively to obtain first sentence information corresponding to the dialogue sentence and second sentence information corresponding to the response sentence.

[0079] S403: Extracting a target corpus corresponding to a preset corpus label from the first sentence information and the second sentence information.

[0080] The description of S401 - S303 can be specifically referred to the above embodiment, which will not be repeated here.

[0081] S404: storing the target corpus corresponding to the preset corpus tag in the big data processing platform, and configuring the preset corpus tag as an index item corresponding to the target corpus.

[0082] After extracting the target corpus corresponding to the preset corpus label from the first sentence information and the second sentence information, the target corpus corresponding to the preset corpus label can be stored in the big data processing platform, and the preset corpus label can be configured as an index item corresponding to the target corpus.

[0083] Among them, the big data processing platform is a data processing platform that completes various business needs by processing tasks such as massive data storage, calculation, and real-time calculation of streaming data.

[0084] Optionally, in order to realize the storage of the target corpus in this embodiment, the big data processing platform described in this embodiment can be a big data platform built on the basis of a data warehouse tool (hive). Of course, you can also build a data storage platform yourself to store the target corpus, and there is no restriction on this.

[0085] Among them, the data warehouse tool (hive) is a data warehouse tool based on Hadoop, which is used for data extraction, transformation and loading. It is a mechanism for storing, querying and analyzing large-scale data stored in Hadoop. The hive data warehouse tool can map structured data files into a database table and provide structured query language query functions, which can convert query statements into mapping functions to execute corresponding business.

[0086] Among them, the index can provide a pointer to the corpus stored in a specified column of a table, and the index item is equivalent to a pointer list of a collection of one or several columns of corpus in a table. By configuring the preset corpus tags as corresponding index items, it can assist in quickly locating the target corpus that matches it based on the preset corpus tags, so that the target corpus can be obtained from the big data platform more quickly.

[0087] In this embodiment, since the target corpus corresponding to the preset corpus label is stored in the big data processing platform, and the preset corpus label is configured as an index item corresponding to the target corpus, the target corpus can be effectively expanded, so that the corpus acquisition user can obtain higher quality corpus during the corpus acquisition process, and the corpus acquisition user is supported to index and locate the matching target corpus based on the preset corpus label, thereby improving the quality of corpus acquisition and the convenience of corpus acquisition.

[0088] S405: providing a corpus acquisition interface, which includes: at least one candidate corpus tag, a time range for corpus acquisition, and the amount of corpus.

[0089] Among them, the corpus acquisition device can provide an operation interface, which can be called a corpus acquisition interface. The corpus acquisition interface displays some corpus tags that can be selected by the user, and the corpus tags that can be selected by the user can be called candidate corpus tags.

[0090] like Figure 5 As shown, Figure 5 It is a schematic diagram of a corpus acquisition interface according to an embodiment of the present disclosure.

[0091] After the target corpus corresponding to the preset corpus tag is stored in the big data processing platform and the preset corpus tag is configured as an index item corresponding to the target corpus, a corpus acquisition interface can be provided. The corpus acquisition interface includes: at least one candidate corpus tag, a time range for corpus acquisition, and the number of corpora. Then, the corpus acquisition user can be supported to create a corresponding corpus acquisition request in the corpus acquisition interface, thereby realizing the corresponding corpus acquisition task.

[0092] Optionally, in some embodiments, it is also possible to support configuring a corpus acquisition device to communicate data with an artificial intelligence model algorithm platform. The corpus acquisition device can provide a corresponding corpus acquisition interface to the corpus acquisition user. When the corpus acquisition interface receives a corpus acquisition request input by the user, the corpus acquisition request can be sent to the artificial intelligence model algorithm platform to obtain the required target corpus.

[0093] S406: receiving a corpus configuration request based on the corpus acquisition interface, taking the candidate corpus tag selected by the corpus configuration request as the corpus tag to be processed, and determining the target time range and target corpus quantity configured by the corpus configuration request.

[0094] After the corpus acquisition interface is provided, the user can select one or more candidate corpus tags as corpus tags to be processed from the corpus acquisition interface, and determine the target time range and target corpus quantity for the corpus configuration request.

[0095] Among them, the request for the user to select candidate corpus tags as corpus tags to be processed from the corpus acquisition interface can be called a corpus configuration request.

[0096] The labels of the corpus to be processed are configured by the user according to the actual corpus acquisition requirements.

[0097] For example, if the user currently needs to extract the corpus corresponding to the dialogue type tag, the user can set the dialogue type tag as the corpus tag to be processed, and the corpus acquisition device can refer to the corpus tag to be processed selected by the user and match it with the corpus tag (preset corpus tag) configured in the big data processing platform, thereby triggering the acquisition of the corpus indexed by the matching preset corpus tag as the target corpus.

[0098] S407: Generate a corpus acquisition request according to the corpus tag to be processed, the target time range, and the target corpus quantity.

[0099] Among them, the target time range and the target corpus quantity are obtained by the user according to actual needs, and the target time range can be used to express the time range corresponding to the corpus that the user needs to extract.

[0100] For example, if the corpus tag to be processed currently selected by the user is a dialogue scene tag, the time range corresponding to the dialogue scene corpus to be extracted is t1-t2, and the number of corpora corresponding to the dialogue scene corpus to be extracted is 30,000, then a corresponding corpus acquisition request can be generated based on the user's above-mentioned corpus extraction requirements.

[0101] Optionally, in some embodiments, after receiving a corpus configuration request based on the corpus acquisition interface, a corpus acquisition request may be generated according to the tag of the corpus to be processed, the target time range, and the target corpus quantity.

[0102] For example, a corpus acquisition request can be generated by a corpus acquisition device, and then the corpus acquisition request can be sent to the artificial intelligence model algorithm platform. The artificial intelligence model algorithm platform parses the corpus acquisition request to obtain the label of the corpus to be processed, the target time range, and the target number of corpora, etc., and refers to the label of the corpus to be processed, the target time range, and the target number of corpora to generate a corresponding corpus query statement, so as to query the target corpus from the artificial intelligence model algorithm platform based on the corpus query statement.

[0103] Thus, by providing a corpus acquisition interface, the corpus acquisition interface includes: at least one candidate corpus tag, a time range for corpus acquisition, and a corpus quantity, a corpus configuration request is received based on the corpus acquisition interface, and the candidate corpus tag selected by the corpus configuration request is used as the corpus tag to be processed, and the target time range and target corpus quantity configured by the corpus configuration request are determined, and the corpus acquisition request is generated according to the corpus tag to be processed, the target time range, and the target corpus quantity, thereby making the corpus acquisition operation of the corpus acquisition user more convenient, making the corpus acquisition method applicable to various application scenarios with different real-time requirements, improving the applicability of the corpus acquisition method, expanding the application scenarios of the corpus acquisition method, and improving the user's corpus acquisition experience.

[0104] S408: Send the corpus acquisition request to the artificial intelligence model algorithm platform.

[0105] S409: The artificial intelligence model algorithm platform determines a preset corpus label corresponding to the corpus label to be processed.

[0106] After the corpus acquisition request is sent to the artificial intelligence model algorithm platform, the artificial intelligence model algorithm platform can determine the preset corpus label corresponding to the corpus label to be processed.

[0107] Optionally, in some embodiments, the similarity value between the label of the corpus to be processed and the preset corpus label can be determined. If the similarity value between the label of the corpus to be processed and the preset corpus label is greater than a preset similarity threshold, it can be determined that there is a mapping relationship between the label of the corpus to be processed and the preset corpus label, and the preset corpus label that has a mapping relationship with the label of the corpus to be processed can be determined as the corresponding preset corpus label.

[0108] S410: Acquire multiple target corpora obtained by corresponding preset corpus tag indexes, and select part of the target corpora from the multiple target corpora according to the target time range and the number of target corpora.

[0109] After determining the preset corpus tag corresponding to the corpus tag to be processed, multiple target corpora indexed by the corresponding preset corpus tag can be obtained, and part of the target corpora can be selected from the multiple target corpora according to the target time range and the number of target corpora.

[0110] For example, multiple target corpora pre-stored in the big data platform can be obtained through a preset corpus tag index, and then part of the target corpora can be selected from the multiple target corpora according to the target time range and the number of target corpora.

[0111] S411: Using part of the target corpus as the corpus obtained based on the corpus acquisition request.

[0112] After obtaining the multiple target corpora corresponding to the preset corpus tag index and selecting part of the target corpora from the multiple target corpora according to the target time range and the number of target corpora, the part of the target corpora can be used as the corpora obtained based on the corpus acquisition request.

[0113] Optionally, in some embodiments, after obtaining part of the target corpus from multiple target corpora according to the corpus acquisition request, the obtained part of the corpus can be written into a record file, and a corresponding file download link can be generated, and then the file download link can be written back into the corpus acquisition record, such as Figure 6 As shown, Figure 6 It is a schematic diagram of the corpus acquisition record interface according to an embodiment of the present disclosure. After the corpus acquisition task is completed, the user can download the record file corresponding to the corpus acquisition request in the corpus acquisition record interface.

[0114] Figure 7 is a flow chart of a method for acquiring corpus according to an embodiment of the present disclosure, such as Figure 7 As shown: Among them, the corpus acquisition method can be composed of two parts of tasks, namely, the target corpus storage task and the corpus acquisition task. When executing the target corpus storage task, the user can propose a dialogue statement through the client of the intelligent dialogue system, and then the server of the intelligent dialogue system will generate a corresponding response statement when receiving the user's dialogue statement, and then the dialogue statement and the response statement can be processed accordingly to obtain the target corpus, and then the above target corpus is pushed to the big data platform for storage, thereby completing the target corpus storage task.

[0115] When executing a corpus acquisition task, the user can input the corpus extraction conditions through the corpus acquisition interface, and then the artificial intelligence model algorithm platform can convert the corpus extraction conditions into a corpus acquisition request. The artificial intelligence model algorithm platform can establish a corpus acquisition task, record the user's query conditions, and connect the query engine (presto) to the big data platform to execute the corpus acquisition request. After the corpus acquisition task is completed, the acquired corpus will be written into a record file, and a download link will be generated and written back to the task record. The user can download the corpus file to complete the corpus acquisition task.

[0116] Thus, in this embodiment, by receiving a corpus acquisition request, the corpus acquisition request includes: a to-be-processed corpus tag, a target time range, and a target corpus quantity, determining a preset corpus tag corresponding to the to-be-processed corpus tag, acquiring multiple target corpora indexed by the corresponding preset corpus tag, and selecting part of the target corpora from the multiple target corpora according to the target time range and the target corpus quantity, and using the part of the target corpora as the corpus acquired based on the corpus acquisition request. Thus, it is possible to accurately select part of the target corpora required by the user from multiple target corpora, thereby effectively improving the efficiency of corpus acquisition, and making the acquired target corpora more suitable for the user's current actual business scenario.

[0117] In this embodiment, by receiving a dialogue sentence and determining a response sentence corresponding to the dialogue sentence, the dialogue sentence and the response sentence are processed respectively to obtain first sentence information corresponding to the dialogue sentence and second sentence information corresponding to the response sentence, and the target corpus corresponding to the preset corpus label is extracted from the first sentence information and the second sentence information, and the target corpus corresponding to the preset corpus label is stored in a big data processing platform, and the preset corpus label is configured as an index item corresponding to the target corpus, so that the target corpus can be effectively expanded, so that the corpus acquisition user can obtain higher quality corpus during the corpus acquisition process, and the corpus acquisition user is supported to index and locate the matching target corpus based on the preset corpus label, thereby improving the quality of corpus acquisition and the convenience of corpus acquisition. By providing a corpus acquisition interface, receiving a corpus configuration request based on the corpus acquisition interface, using the candidate corpus tag selected by the corpus configuration request as the corpus tag to be processed, and determining the target time range and target corpus quantity configured by the corpus configuration request, and generating the corpus acquisition request according to the corpus tag to be processed, the target time range, and the target corpus quantity, the corpus acquisition request can be made more convenient for the corpus acquisition user, so that the corpus acquisition method can be applied to various application scenarios with different real-time requirements, thereby improving the applicability of the corpus acquisition method, expanding the application scenarios of the corpus acquisition method, and improving the user's corpus acquisition experience.

[0118] Figure 8 It is a structural diagram of a corpus acquisition device proposed in an embodiment of the present disclosure.

[0119] like Figure 8 As shown, the corpus acquisition device 80 includes:

[0120] The first receiving module 801 is used to receive a dialogue statement and determine a response statement corresponding to the dialogue statement;

[0121] A first processing module 802, used to process the dialogue sentence and the response sentence respectively to obtain first sentence information corresponding to the dialogue sentence and second sentence information corresponding to the response sentence;

[0122] The extraction module 803 is used to extract the target corpus corresponding to the preset corpus label from the first sentence information and the second sentence information.

[0123] In some embodiments of the present disclosure, Fig. 9 As shown, the extraction module 803 is specifically used for:

[0124] Determining conversation description information according to the first sentence information and the second sentence information;

[0125] If there is a mapping relationship between the dialogue description information and the preset corpus label, a target corpus corresponding to the preset corpus label is extracted from the first sentence information and the second sentence information.

[0126] In some embodiments of the present disclosure, the extraction module 803 is specifically used to:

[0127] After determining the conversation description information according to the first sentence information and the second sentence information, determining an initial corpus label according to the conversation description information;

[0128] If the initial corpus label matches the preset corpus label, it is determined that there is a mapping relationship between the dialogue description information and the preset corpus label.

[0129] In some embodiments of the present disclosure, the extraction module 803 is specifically used to:

[0130] After determining the conversation description information according to the first sentence information and the second sentence information, determining a similarity value between the conversation description information and a tag content of the preset corpus tag;

[0131] If the similarity value is greater than a set threshold, it is determined that there is a mapping relationship between the conversation description information and the preset corpus label.

[0132] In some embodiments of the present disclosure, the corpus acquisition device 80 further includes:

[0133] The storage module 804 is used to store the target corpus corresponding to the preset corpus label in the big data processing platform after extracting the target corpus corresponding to the preset corpus label from the first sentence information and the second sentence information, and configure the preset corpus label as an index item corresponding to the target corpus.

[0134] In some embodiments of the present disclosure, the number of preset corpus tags is multiple, wherein the corpus acquisition device 80 further includes:

[0135] The second receiving module 805 is used to receive a corpus acquisition request, where the corpus acquisition request includes: a corpus tag to be processed, a target time range, and a target corpus quantity;

[0136] A determination module 806 is used to determine a preset corpus label corresponding to the corpus label to be processed;

[0137] The acquisition module 807 is used to acquire a plurality of target corpora obtained by the corresponding preset corpus tag index, and select part of the target corpora from the plurality of target corpora according to the target time range and the number of target corpora;

[0138] The second processing module 808 is configured to use part of the target corpus as the corpus obtained based on the corpus acquisition request.

[0139] In some embodiments of the present disclosure, the corpus acquisition device 80 further includes:

[0140] A providing module 809 is used to provide a corpus acquisition interface, which includes: at least one candidate corpus tag, a time range for corpus acquisition, and a corpus quantity;

[0141] The third receiving module 810 is used to receive a corpus configuration request based on the corpus acquisition interface, and use the candidate corpus tag selected by the corpus configuration request as the corpus tag to be processed, and determine the target time range and target corpus quantity configured in the corpus configuration request;

[0142] The generating module 811 is used to generate a corpus acquisition request according to the corpus tag to be processed, the target time range, and the target corpus quantity.

[0143] In some embodiments of the present disclosure, the preset corpus tags include any one or more combinations of the following:

[0144] Dialogue type labels, dialogue scene labels, and dialogue keyword labels.

[0145] With the above Figures 1 to 7 Corresponding to the corpus acquisition method provided in the embodiment, the present disclosure also provides a corpus acquisition device. Figures 1 to 7 The corpus acquisition method provided in the embodiment corresponds to the corpus acquisition method, so the implementation of the corpus acquisition method is also applicable to the corpus acquisition device provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0146] In order to implement the above embodiments, the present disclosure further proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the corpus acquisition method proposed in the above embodiments of the present disclosure is implemented.

[0147] In order to implement the above embodiments, the present disclosure further proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the corpus acquisition method proposed in the above embodiments of the present disclosure is implemented.

[0148] In order to implement the above embodiments, the present disclosure further proposes a computer program product. When an instruction processor in the computer program product is executed, the corpus acquisition method proposed in the above embodiments of the present disclosure is executed.

[0149] Fig.10 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Fig.10 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0150] like Fig.10 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0151] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0152] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0153] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Fig.10 Not shown, often called a "hard drive").

[0154] although Fig.10 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present disclosure.

[0155] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0156] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0157] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing the corpus acquisition method mentioned in the above embodiment.

[0158] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0159] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0160] It should be noted that, in the description of the present disclosure, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "plurality" is two or more.

[0161] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0162] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0163] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0164] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0165] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0166] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0167] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. A corpus acquisition method, characterized in that: include: receiving a dialogue statement and determining a response statement corresponding to the dialogue statement; Processing the dialogue sentence and the response sentence respectively to obtain first sentence information corresponding to the dialogue sentence and second sentence information corresponding to the response sentence; Extracting a target corpus corresponding to a preset corpus tag from the first sentence information and the second sentence information, wherein the preset corpus tag is pre-set as a embedding tag in the intelligent dialogue system, and the preset corpus tag includes any one or more combinations of the following: Dialogue type tags, dialogue scene tags, dialogue keyword tags; The target corpus corresponding to the preset corpus label is stored in a big data processing platform, and the preset corpus label is configured as an index item corresponding to the target corpus; The number of the preset corpus tags is multiple, and also includes: Receive a corpus acquisition request, the corpus acquisition request including: a label of the corpus to be processed, a target time range, and a target corpus quantity; Determine a preset corpus label corresponding to the corpus label to be processed; Acquire multiple target corpora obtained by the corresponding preset corpus tag index, and select part of the target corpora from the multiple target corpora according to the target time range and the target corpus quantity; The part of the target corpus is used as the corpus obtained based on the corpus acquisition request.

2. The method according to claim 1, characterized in that The extracting the target corpus corresponding to the preset corpus label from the first sentence information and the second sentence information includes: Determining conversation description information according to the first sentence information and the second sentence information; If there is a mapping relationship between the dialogue description information and the preset corpus label, a target corpus corresponding to the preset corpus label is extracted from the first sentence information and the second sentence information.

3. The method according to claim 2, characterized in that After determining the dialogue description information according to the first sentence information and the second sentence information, the method further includes: Determining an initial corpus label according to the conversation description information; If the initial corpus label matches the preset corpus label, it is determined that there is a mapping relationship between the dialogue description information and the preset corpus label.

4. The method according to claim 2, characterized in that After determining the dialogue description information according to the first sentence information and the second sentence information, the method further includes: Determining a similarity value between the conversation description information and the tag content of the preset corpus tag; If the similarity value is greater than a set threshold, it is determined that there is a mapping relationship between the conversation description information and the preset corpus label.

5. The method according to claim 1, characterized in that Also includes: Providing a corpus acquisition interface, the corpus acquisition interface including: at least one candidate corpus tag, a time range for corpus acquisition, and a corpus quantity; Receiving a corpus configuration request based on the corpus acquisition interface, using the candidate corpus tag selected by the corpus configuration request as the corpus tag to be processed, and determining a target time range and a target corpus quantity configured by the corpus configuration request; The corpus acquisition request is generated according to the to-be-processed corpus label, the target time range, and the target corpus quantity.

6. A corpus acquisition device, characterized in that: include: A first receiving module, configured to receive a dialogue statement and determine a response statement corresponding to the dialogue statement; A first processing module, used for processing the dialogue sentence and the response sentence respectively to obtain first sentence information corresponding to the dialogue sentence and second sentence information corresponding to the response sentence; An extraction module is used to extract a target corpus corresponding to a preset corpus tag from the first sentence information and the second sentence information, wherein the preset corpus tag is pre-set as a embedding tag in the intelligent dialogue system, and the preset corpus tag includes any one or more of the following combinations: Dialogue type tags, dialogue scene tags, dialogue keyword tags; A storage module, configured to, after extracting the target corpus corresponding to the preset corpus label from the first sentence information and the second sentence information, store the target corpus corresponding to the preset corpus label in a big data processing platform, and configure the preset corpus label as an index item corresponding to the target corpus; The number of the preset corpus tags is multiple, and also includes: A second receiving module is used to receive a corpus acquisition request, wherein the corpus acquisition request includes: a label of the corpus to be processed, a target time range, and a target corpus quantity; A determination module, used to determine a preset corpus label corresponding to the corpus label to be processed; An acquisition module, used for acquiring a plurality of target corpora obtained by the corresponding preset corpus tag index, and selecting part of the target corpora from the plurality of target corpora according to the target time range and the number of the target corpora; The second processing module is used to use the part of the target corpus as the corpus obtained based on the corpus acquisition request.

7. The device according to claim 6, characterized in that The extraction module is specifically used for: Determining conversation description information according to the first sentence information and the second sentence information; If there is a mapping relationship between the dialogue description information and the preset corpus label, a target corpus corresponding to the preset corpus label is extracted from the first sentence information and the second sentence information.

8. The device according to claim 7, characterized in that The extraction module is further used for: After determining the conversation description information according to the first sentence information and the second sentence information, determining an initial corpus label according to the conversation description information; If the initial corpus label matches the preset corpus label, it is determined that there is a mapping relationship between the dialogue description information and the preset corpus label.

9. The device according to claim 7, characterized in that The extraction module is further used for: After determining the conversation description information according to the first sentence information and the second sentence information, determining a similarity value between the conversation description information and a tag content of the preset corpus tag; If the similarity value is greater than a set threshold, it is determined that there is a mapping relationship between the conversation description information and the preset corpus label.

10. The device according to claim 6, characterized in that Also includes: A module is provided, for providing a corpus acquisition interface, wherein the corpus acquisition interface includes: at least one candidate corpus tag, a time range for corpus acquisition, and a corpus quantity; A third receiving module is used to receive a corpus configuration request based on the corpus acquisition interface, and use the candidate corpus tag selected by the corpus configuration request as the corpus tag to be processed, and determine the target time range and target corpus quantity configured by the corpus configuration request; A generating module is used to generate the corpus acquisition request according to the to-be-processed corpus label, the target time range, and the target corpus quantity.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

Images