Text library construction method and device, and electronic equipment
By automatically expanding the text library in intent recognition scenarios and utilizing vectorization processing and clustering algorithms, the problems of low efficiency in text library construction and low accuracy in intent recognition are solved, achieving efficient text library construction and improved accuracy of intent recognition results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHENGDU MEITA WORLD TECH CO LTD
- Filing Date
- 2024-04-22
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies are inefficient in building text libraries and cannot obtain sufficiently comprehensive text samples, resulting in low accuracy of intent recognition results.
By identifying multiple intent categories related to the intent recognition scenario, question text and answer text are determined for each intent category. Target text is automatically expanded from the full text to build a text library. Vectorization processing and clustering algorithms are used to improve the linguistic diversity of the text library.
It improved the efficiency of text library construction and significantly improved the accuracy of intent recognition results through the expanded text library.
Smart Images
Figure CN118467727B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus and electronic device for constructing a text library. Background Technology
[0002] In some scenarios, it is necessary to perform intent recognition on the user's question text. In order to achieve accurate intent classification, it is usually necessary to build a text library in advance to perform intent recognition based on the pre-built text library.
[0003] Currently, when building a text library, the main approach is to identify text samples from massive amounts of chat logs and then manually label the text samples with intent categories, thereby forming a text library of a certain scale.
[0004] However, using the above-mentioned manual method for intent category labeling results in low efficiency in building the text library and an inability to obtain sufficiently comprehensive text samples for building the text library. Consequently, the accuracy of intent recognition results is low when the intent recognition is performed based on the constructed text library. Summary of the Invention
[0005] This application provides a method, apparatus, and electronic device for constructing a text library. When constructing the text library, the construction efficiency can be improved, and when performing intent recognition based on the constructed text library, the accuracy of the intent recognition results can be effectively improved.
[0006] This application provides a method for constructing a text library, including:
[0007] Identify multiple intent categories related to the intent recognition scenario;
[0008] For each intent category, determine multiple question texts corresponding to the intent category and answer texts corresponding to each question text;
[0009] Based on the answer text corresponding to each of the aforementioned question texts, multiple target texts related to the intent category are determined from the full text.
[0010] Based on the multiple question texts, the answer texts corresponding to each question text, and the multiple target texts, a text library corresponding to the intent category is constructed.
[0011] According to a method for constructing a text library provided in this application, determining multiple target texts related to the intent category from the full text based on the answer texts corresponding to each question text includes:
[0012] Based on the answer text corresponding to each of the aforementioned question texts, determine the target center vector corresponding to the intent category;
[0013] Clustering is performed on the entire text to obtain multiple clustering results;
[0014] Determine the similarity between the target center vector and the center vectors corresponding to each of the clustering results;
[0015] The full text included in the clustering result corresponding to the maximum similarity is determined as the multiple target texts.
[0016] According to the text library construction method provided in this application, determining the target center vector corresponding to the intent category based on the answer text corresponding to each question text includes:
[0017] The answer texts corresponding to each of the aforementioned question texts are vectorized to obtain the center vectors of each of the aforementioned answer texts;
[0018] The target center vector is determined based on the average value of the center vectors of each of the answer texts.
[0019] According to the method for constructing a text library provided in this application, the method further includes:
[0020] Obtain the text to be recognized;
[0021] Determine the similarity between the text to be identified and multiple texts in the text library corresponding to each intent category;
[0022] The intent recognition result of the text to be identified is determined based on the maximum similarity.
[0023] According to the text library construction method provided in this application, determining the intent recognition result of the text to be recognized based on the maximum similarity includes:
[0024] When the maximum similarity is greater than or equal to the lower limit of the similarity threshold and less than the upper limit of the similarity threshold, the matching result of the text to be identified and the text corresponding to the maximum similarity is determined based on at least two association feature information between the text to be identified and the text corresponding to the maximum similarity, and the at least two association feature information include at least two of the following: common substring length, synonyms and near-synonyms, and the number of shared words.
[0025] Based on each of the matching results, the intent recognition result of the text to be recognized is determined.
[0026] According to the text library construction method provided in this application, determining the intent recognition result of the text to be recognized based on each of the matching results includes:
[0027] If all the matching results are the same, the matching result is determined as the intent recognition result of the text to be recognized;
[0028] If all the matching results are not identical, the matching result of the highest priority associated feature information is determined as the intent recognition result of the text to be recognized, based on the priority of the associated feature information.
[0029] This application also provides a text library construction apparatus, comprising:
[0030] The first processing unit is used to determine multiple intent categories related to the intent recognition scenario;
[0031] The second processing unit is configured to determine, for each intent category, multiple question texts corresponding to the intent category and answer texts corresponding to each question text;
[0032] The third processing unit is used to determine multiple target texts related to the intent category from the full text based on the answer texts corresponding to each question text;
[0033] The construction unit is used to construct a text library corresponding to the intent category based on the plurality of question texts, the answer texts corresponding to each question text, and the plurality of target texts.
[0034] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for constructing a text library as described in any of the preceding claims.
[0035] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for constructing a text library as described in any of the preceding claims.
[0036] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method for constructing a text library as described in any of the preceding claims.
[0037] This application provides a method, apparatus, and electronic device for constructing a text library. When constructing the text library, multiple intent categories related to the intent recognition scenario are first determined. For each intent category, multiple question texts and corresponding answer texts are determined. Based on the answer texts, multiple target texts related to the intent category are determined from the full text. Based on the multiple question texts, their corresponding answer texts, and the multiple target texts, a text library corresponding to the intent category is constructed. This automatic expansion of the text corresponding to the intent category using multiple target texts not only effectively improves the efficiency of text library construction but also allows the expanded text to better cover linguistic diversity. This results in improved accuracy of intent recognition results when performing intent recognition based on the constructed text library. Attached Figure Description
[0038] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 A flowchart illustrating a method for constructing a text library as provided in an embodiment of this application;
[0040] Figure 2 A flowchart illustrating a method for determining multiple target texts related to intent categories from a full text dataset, as provided in this application embodiment;
[0041] Figure 3 This application provides a schematic flowchart of a method for intent recognition of text to be recognized, as illustrated in an embodiment of the present application.
[0042] Figure 4 A schematic diagram of a text library construction apparatus provided in an embodiment of this application;
[0043] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0044] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0045] In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0046] The technical solutions provided in this application can be adapted to text intent recognition scenarios. In some scenarios, it is necessary to recognize the intent of a user's question text. In order to achieve accurate intent classification, it is usually necessary to pre-build a text library to perform intent recognition based on the pre-built text library.
[0047] Currently, the construction of a text library mainly involves identifying text samples from massive amounts of chat logs and manually labeling these samples with intent categories to form a text library of a certain size. However, this manual intent category labeling method results in low efficiency in building the text library and fails to obtain sufficiently comprehensive text samples for construction, leading to low accuracy in intent recognition results when the intent recognition is performed based on the constructed text library.
[0048] To improve the efficiency of text library construction and enhance the accuracy of intent recognition results based on the constructed text library, this application provides a method for constructing a text library. The following specific embodiments will illustrate this method in detail. It is understood that these specific embodiments can be combined with each other, and similar concepts or processes may not be repeated in some embodiments.
[0049] Figure 1 This application provides a flowchart illustrating a method for constructing a text library, which can be executed by software and / or hardware devices. For example, please refer to... Figure 1 As shown, the method may include:
[0050] S101. Identify multiple intent categories related to the intent recognition scenario.
[0051] For example, in the embodiments of this application, the intent recognition scenario can be intent recognition under a search engine, intent recognition under smart IoT devices, or intent recognition scenario under a dialogue system, etc., and can be set according to actual needs.
[0052] Taking the intent recognition scenario in a dialogue system as an example, the multiple intent categories related to this intent recognition scenario may include e-commerce category, ticket purchase category, weather query category, restaurant query category, or shopping guide category, etc., which can be set according to actual needs.
[0053] S102. For each intent category, determine the multiple question texts corresponding to the intent category and the answer texts corresponding to each question text.
[0054] The number of multiple question texts can be set according to actual needs. For example, in this embodiment of the application, the number of multiple question texts can be about ten.
[0055] For each intent category, multiple question texts corresponding to the intent category can be determined first. When determining the answer text corresponding to each question text, for example, the answer text corresponding to the question text can be queried from the historical chat history, and the answer text can be recalled to obtain the answer text corresponding to the question text.
[0056] After obtaining the answer texts corresponding to each question text, multiple target texts related to the intent category can be determined from the full text based on the answer texts corresponding to each question text. That is, the following S103 is executed to expand the text corresponding to the intent category using the determined multiple target texts. This not only realizes the automatic expansion of text without the need for the above-mentioned manual method of intent category labeling, but also the expanded text can better cover the diversity of languages, so that when performing intent recognition based on the constructed text library, the accuracy of intent recognition results can be effectively improved.
[0057] S103. Based on the answer text corresponding to each question text, determine multiple target texts related to the intent category from the full text.
[0058] The full text usually refers to all the text content in a collection or data source, and is often used for analysis and learning.
[0059] Taking the intent recognition scenario in a dialogue system as an example, the multiple target texts related to the intent recognition scenario may include texts related to e-commerce, ticket purchase, weather query, restaurant query, or shopping guide, etc. The specific settings can be made according to actual needs. Here, this application embodiment does not make further limitations.
[0060] S104. Based on multiple question texts, the answer texts corresponding to each question text, and multiple target texts, construct a text library corresponding to the intent category.
[0061] The constructed text library includes multiple question texts, corresponding answer texts for each question text, and multiple target texts.
[0062] As can be seen from the embodiments of this application, when constructing the text library, multiple intent categories related to the intent recognition scenario can be determined first; for each intent category, multiple question texts corresponding to the intent category and answer texts corresponding to each question text are determined; based on the answer texts corresponding to each question text, multiple target texts related to the intent category are determined from the full text; based on the multiple question texts, the answer texts corresponding to each question text, and the multiple target texts, a text library corresponding to the intent category is constructed. This automatic expansion of the texts corresponding to the intent category using multiple target texts not only effectively improves the efficiency of text library construction but also allows the expanded texts to better cover language diversity, thereby effectively improving the accuracy of intent recognition results when performing intent recognition based on the constructed text library.
[0063] Based on the above Figure 1 The illustrated embodiment, in order to facilitate understanding of how, in S103 above, multiple target texts related to the intent category are determined from the full text based on the answer text corresponding to each question text, will be explained below. Figure 2 The embodiments shown are described in detail below.
[0064] Figure 2 This application provides a flowchart illustrating a method for determining multiple target texts related to an intent category from a full text dataset. This method can also be executed by software and / or hardware devices. For example, please refer to [link to relevant documentation]. Figure 2 As shown, the method may include:
[0065] S201. Based on the answer text corresponding to each question text, determine the target center vector corresponding to the intent category.
[0066] For example, in the embodiments of this application, when determining the target center vector corresponding to the intent category based on the answer text corresponding to each question text, the answer text corresponding to each question text can be vectorized to obtain the center vector of each answer text; and the target center vector is determined based on the average value of the center vectors of each answer text.
[0067] For example, in the embodiments of this application, when vectorizing the answer text corresponding to the question text, BCE text vectorization technology can be used to vectorize the answer text corresponding to the question text. Alternatively, other text vectorization technologies can be used to vectorize the answer text corresponding to the question text, such as TF-IDF (Term Frequency-Inverse Document Frequency) technology or Word2Vec technology, etc. The specific settings can be made according to actual needs.
[0068] For example, in this embodiment of the application, when determining the target center vector corresponding to the intent category based on the average value of the center vectors of each answer text, the average value of the center vectors of each answer text can be directly determined as the target center vector corresponding to the intent category; alternatively, the average value of the center vectors of each answer text can be processed, such as normalized, and the processing result can be determined as the target center vector corresponding to the intent category. The specific settings can be made according to actual needs.
[0069] S202. Cluster the entire text to obtain multiple clustering results.
[0070] For example, in the embodiments of this application, when clustering the full text to obtain multiple clustering results, the BCE text vectorization technology can be used first to vectorize the full text to obtain the text vectors corresponding to each of the full texts; then, a clustering algorithm, such as the DBScan unsupervised clustering algorithm, or other clustering algorithms, can be used to cluster the text vectors corresponding to each of the full texts to obtain multiple clustering results.
[0071] It is understood that in the embodiments of this application, there is no specific order between S201 and S202. The embodiments of this application are only used as an example of executing S201 first and then S202, but this does not mean that the embodiments of this application are limited to this.
[0072] After combining S201 and S202 above, and having determined the target center vectors corresponding to the intent categories and the results of each clustering, the following S203 can be further executed:
[0073] S203. Determine the similarity between the target center vector and the center vectors corresponding to each clustering result.
[0074] The center vector corresponding to the clustering result can be understood as the weighted average vector of multiple text vectors included in the clustering result.
[0075] For example, in the embodiments of this application, when determining the similarity between the target center vector and the center vector corresponding to the clustering result, the cosine distance between the target center vector and the center vector corresponding to the clustering result can be calculated first, and the similarity between the target center vector and the center vector corresponding to the clustering result can be determined based on the calculated cosine distance.
[0076] Generally, the smaller the cosine distance, the larger the corresponding similarity value; the larger the cosine distance, the smaller the corresponding similarity value.
[0077] S204. The full text included in the clustering results corresponding to the maximum similarity is identified as multiple target texts.
[0078] As can be seen, in this embodiment of the application, when automatically expanding the text corresponding to the intent category, the target center vector corresponding to the intent category can be determined based on the answer text corresponding to each question text; and the entire text can be clustered to obtain multiple clustering results; then the similarity between the target center vector and the center vector corresponding to each clustering result can be determined respectively, and the entire text included in the clustering result corresponding to the maximum similarity can be determined as multiple target texts. In this way, the text corresponding to the intent category can be automatically expanded based on multiple target texts. When constructing a text library corresponding to the intent category based on multiple question texts, the answer text corresponding to each question text, and multiple target texts, not only can the construction efficiency of the text library be effectively improved, but the expanded text can also better cover the diversity of languages, so that when performing intent recognition based on the constructed text library, the accuracy of the intent recognition result can be effectively improved.
[0079] Based on any of the above embodiments, after constructing a text library corresponding to each intent category among multiple intent categories related to the intent recognition scenario, the text library corresponding to each intent category can be used as the basis for intent recognition and used to perform intent recognition on the text to be recognized. For details, please refer to the following... Figure 3 The example shown.
[0080] Figure 3 This is a schematic flowchart illustrating a method for intent recognition of text to be recognized, provided in an embodiment of this application. This method can also be executed by software and / or hardware devices. For example, please refer to... Figure 3 As shown, the method may include:
[0081] S301. Obtain the text to be recognized.
[0082] For example, in the embodiments of this application, the text to be identified can be obtained in a variety of ways, such as receiving the text to be identified input by the user, receiving the text to be identified sent by other electronic devices, or obtaining the text to be identified from a third-party database, etc. The specific settings can be configured according to actual needs.
[0083] S302. Determine the similarity between the text to be identified and multiple texts in the text library corresponding to each intent category.
[0084] For example, in the embodiments of this application, when determining the similarity between the text to be identified and multiple texts in the text library corresponding to the intent category, the text to be identified can first be vectorized to obtain the text vector corresponding to the text to be identified; then, the cosine distance between the text vector corresponding to the text to be identified and the text vectors corresponding to the multiple texts in the text library is determined, and the similarity between the text to be identified and the multiple texts in the text library is determined based on the cosine distance between the text vector corresponding to the text to be identified and the text vectors corresponding to the multiple texts in the text library.
[0085] S303. Based on the maximum similarity, determine the intent recognition result of the text to be recognized.
[0086] For example, in the embodiments of this application, when determining the intent recognition result of the text to be recognized based on the maximum similarity, the following three possible scenarios may be included:
[0087] In one possible scenario, the maximum similarity is less than the lower limit of the similarity threshold.
[0088] If the maximum similarity is less than the lower limit of the similarity threshold, it means that the text corresponding to the maximum similarity is not related text to the text to be identified, and the corresponding intent recognition result is that no related text to the text to be identified was found.
[0089] In another possible scenario, the maximum similarity is greater than or equal to the lower limit of the similarity threshold, but less than the upper limit of the similarity threshold.
[0090] If the maximum similarity is greater than or equal to the lower limit of the similarity threshold, but less than the upper limit of the similarity threshold, in order to ensure the accuracy of the intent recognition result, the matching result of the text to be recognized and the text corresponding to the maximum similarity can be determined based on at least two related feature information between the text to be recognized and the text corresponding to the maximum similarity. Among them, the at least two related feature information include at least two of the following: the length of the common substring, synonyms and near-synonyms, and the number of shared words. Based on each matching result, the intent recognition result of the text to be recognized is determined.
[0091] When the associated feature information includes the length of the common substring, the matching result between the text to be identified and the text corresponding to the maximum similarity is determined based on the length of the common substring between the text to be identified and the text corresponding to the maximum similarity. This can be achieved by detecting the length of the longest common substring between the two texts. If the length of the longest non-contiguous common substring between the two texts is less than a length threshold, it indicates that the text corresponding to the maximum similarity is not a related text of the text to be identified, and the corresponding intent recognition result is that no related text of the text to be identified was found. Conversely, if the length of the longest non-contiguous common substring between the two texts is greater than or equal to the length threshold, it indicates that the text corresponding to the maximum similarity is a related text of the text to be identified, and the corresponding intent recognition result is the text corresponding to the maximum similarity.
[0092] When the associated feature information includes synonyms and near-synonyms, to determine the matching result between the text to be identified and the text corresponding to the maximum similarity based on the synonyms and near-synonyms between the two texts, the Hanlp package can be used to perform dependency parsing on the two texts to find the core word HEAD, i.e., synonyms. If they are different, near-synonym detection is performed to prevent cases where the literal meanings are different but the semantics are the same. If there are no multiple synonyms or near-synonyms between the two texts, it means that the text corresponding to the maximum similarity is not a related text of the text to be identified, and the corresponding intent recognition result is that no related text of the text to be identified was found. Conversely, if there are multiple synonyms or near-synonyms between the two texts, it means that the text corresponding to the maximum similarity is a related text of the text to be identified, and the corresponding intent recognition result is the text corresponding to the maximum similarity.
[0093] When the associated feature information includes the number of shared words, the matching result between the text to be identified and the text corresponding to the maximum similarity is determined based on the number of shared words between them. This involves detecting unordered word co-occurrence between the two texts. If the number of shared words is less than a threshold, it indicates that the text corresponding to the maximum similarity is not a related text of the text to be identified, and the corresponding intent recognition result is that no related text of the text to be identified was found. Conversely, if the number of shared words is greater than or equal to the threshold, it indicates that the text corresponding to the maximum similarity is a related text of the text to be identified, and the corresponding intent recognition result is the text corresponding to the maximum similarity.
[0094] For example, in the embodiments of this application, when determining the intent recognition result of the text to be recognized based on each matching result, if all matching results are the same, the matching result is determined as the intent recognition result of the text to be recognized; if all matching results are not the same, the matching result of the associated feature information corresponding to the highest priority is determined as the intent recognition result of the text to be recognized according to the priority of the associated feature information.
[0095] The order of priority from highest to lowest is as follows: length of common substring, synonyms and near-synonyms, and number of shared words.
[0096] For example, taking the associated feature information including the length of the common substring, the number of synonyms and near-synonyms, and the number of shared words as an example, when the intent recognition results determined based on the length of the common substring, the number of synonyms and near-synonyms, and the number of shared words are not all the same, since the length of the common substring has the highest priority, the intent recognition result corresponding to the length of the common substring can be determined as the intent recognition result of the text to be recognized.
[0097] As can be seen, in this possible scenario, the intent recognition result of the text to be recognized can be further determined based on at least two related feature information between the text to be recognized and the text corresponding to the maximum similarity. This completes the multi-angle detection of the text to be recognized, and cleverly performs a secondary judgment on the results of the text vector correlation recall, preventing false positives and false negatives, thereby further improving the accuracy of the intent recognition result.
[0098] In another possible scenario, the maximum similarity is greater than or equal to the upper limit of the similarity threshold.
[0099] If the maximum similarity is greater than or equal to the upper limit of the similarity threshold, it means that the text corresponding to the maximum similarity is related to the text to be identified, and the corresponding intent recognition result is the text corresponding to the maximum similarity.
[0100] The text library construction apparatus provided in this application will now be described. The text library construction apparatus described below can be referred to in correspondence with the text library construction method described above.
[0101] Figure 4 A schematic diagram of a text library construction apparatus provided in this application embodiment is shown below. For example, please refer to... Figure 4 As shown, the text library construction apparatus 40 may include:
[0102] The first processing unit 401 is used to determine multiple intent categories related to the intent recognition scenario;
[0103] The second processing unit 402 is configured to determine, for each intent category, multiple question texts corresponding to the intent category and answer texts corresponding to each question text;
[0104] The third processing unit 403 is used to determine multiple target texts related to the intent category from the full text based on the answer texts corresponding to each question text;
[0105] The construction unit 404 is used to construct a text library corresponding to the intent category based on the plurality of question texts, the answer texts corresponding to each question text, and the plurality of target texts.
[0106] For example, in this embodiment of the application, the third processing unit 403 is configured to determine multiple target texts related to the intent category from the full text based on the answer texts corresponding to each of the question texts, including:
[0107] Based on the answer text corresponding to each of the aforementioned question texts, determine the target center vector corresponding to the intent category;
[0108] Clustering is performed on the entire text to obtain multiple clustering results;
[0109] Determine the similarity between the target center vector and the center vectors corresponding to each of the clustering results;
[0110] The full text included in the clustering result corresponding to the maximum similarity is determined as the multiple target texts.
[0111] For example, in this embodiment of the application, the third processing unit 403 is used to determine the target center vector corresponding to the intent category based on the answer text corresponding to each of the question texts, including:
[0112] The answer texts corresponding to each of the aforementioned question texts are vectorized to obtain the center vectors of each of the aforementioned answer texts;
[0113] The target center vector is determined based on the average value of the center vectors of each of the answer texts.
[0114] For example, in an embodiment of this application, the text library construction apparatus further includes:
[0115] The acquisition unit is used to acquire the text to be recognized;
[0116] The fourth processing unit is used to determine the similarity between the text to be identified and multiple texts in the text library corresponding to each intent category;
[0117] The fifth processing unit is used to determine the intent recognition result of the text to be recognized based on the maximum similarity.
[0118] For example, in an embodiment of this application, the fifth processing unit is used to determine the intent recognition result of the text to be recognized based on the maximum similarity, including:
[0119] When the maximum similarity is greater than or equal to the lower limit of the similarity threshold and less than the upper limit of the similarity threshold, the matching result of the text to be identified and the text corresponding to the maximum similarity is determined based on at least two association feature information between the text to be identified and the text corresponding to the maximum similarity, and the at least two association feature information include at least two of the following: common substring length, synonyms and near-synonyms, and the number of shared words.
[0120] Based on each of the matching results, the intent recognition result of the text to be recognized is determined.
[0121] For example, in an embodiment of this application, the fifth processing unit is configured to determine the intent recognition result of the text to be recognized based on each of the matching results, including:
[0122] If all the matching results are the same, the matching result is determined as the intent recognition result of the text to be recognized;
[0123] If all the matching results are not identical, the matching result of the highest priority associated feature information is determined as the intent recognition result of the text to be recognized, based on the priority of the associated feature information.
[0124] The text library construction apparatus 40 provided in this application embodiment can execute the technical solution of the text library construction method in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the text library construction method. Please refer to the implementation principle and beneficial effects of the text library construction method. It will not be repeated here.
[0125] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the above-mentioned text library construction method, which includes: determining multiple intent categories related to the intent recognition scenario; for each intent category, determining multiple question texts corresponding to the intent category and answer texts corresponding to each question text; based on the answer texts corresponding to each question text, determining multiple target texts related to the intent category from the full text; and constructing a text library corresponding to the intent category based on the multiple question texts, the answer texts corresponding to each question text, and the multiple target texts.
[0126] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0127] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the above-mentioned method for constructing a text library. The method includes: determining multiple intent categories related to an intent recognition scenario; for each intent category, determining multiple question texts corresponding to the intent category and answer texts corresponding to each question text; based on the answer texts corresponding to each question text, determining multiple target texts related to the intent category from the full text; and constructing a text library corresponding to the intent category based on the multiple question texts, the answer texts corresponding to each question text, and the multiple target texts.
[0128] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the above-mentioned method for constructing a text library. The method includes: determining multiple intent categories related to an intent recognition scenario; for each intent category, determining multiple question texts corresponding to the intent category and answer texts corresponding to each question text; based on the answer texts corresponding to each question text, determining multiple target texts related to the intent category from the full text; and constructing a text library corresponding to the intent category based on the multiple question texts, the answer texts corresponding to each question text, and the multiple target texts.
[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for constructing a text library, characterized in that, include: Identify multiple intent categories related to the intent recognition scenario; For each intent category, determine multiple question texts corresponding to the intent category and answer texts corresponding to each question text; Based on the answer text corresponding to each of the aforementioned question texts, multiple target texts related to the intent category are determined from the full text. Based on the multiple question texts, the answer texts corresponding to each question text, and the multiple target texts, a text library corresponding to the intent category is constructed; The step of determining multiple target texts related to the intent category from the full text based on the answer texts corresponding to each of the question texts includes: Based on the answer text corresponding to each of the aforementioned question texts, determine the target center vector corresponding to the intent category; Clustering is performed on the entire text to obtain multiple clustering results; Determine the similarity between the target center vector and the center vectors corresponding to each of the clustering results; The full text included in the clustering result corresponding to the maximum similarity is determined as the multiple target texts.
2. The method according to claim 1, characterized in that, The step of determining the target center vector corresponding to the intent category based on the answer text corresponding to each of the question texts includes: The answer texts corresponding to each of the aforementioned question texts are vectorized to obtain the center vectors of each of the aforementioned answer texts; The target center vector is determined based on the average value of the center vectors of each of the answer texts.
3. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain the text to be recognized; Determine the similarity between the text to be identified and multiple texts in the text library corresponding to each intent category; The intent recognition result of the text to be identified is determined based on the maximum similarity.
4. The method according to claim 3, characterized in that, The determination of the intent recognition result of the text to be identified based on the maximum similarity includes: When the maximum similarity is greater than or equal to the lower limit of the similarity threshold and less than the upper limit of the similarity threshold, the matching result of the text to be identified and the text corresponding to the maximum similarity is determined based on at least two association feature information between the text to be identified and the text corresponding to the maximum similarity, and the at least two association feature information include at least two of the following: common substring length, synonyms and near-synonyms, and the number of shared words. Based on each of the matching results, the intent recognition result of the text to be recognized is determined.
5. The method according to claim 4, characterized in that, The step of determining the intent recognition result of the text to be recognized based on each of the matching results includes: If all the matching results are the same, the matching result is determined as the intent recognition result of the text to be recognized; If all the matching results are not identical, the matching result of the highest priority associated feature information is determined as the intent recognition result of the text to be recognized, based on the priority of the associated feature information.
6. A text library construction apparatus, characterized in that, include: The first processing unit is used to determine multiple intent categories related to the intent recognition scenario; The second processing unit is configured to determine, for each intent category, multiple question texts corresponding to the intent category and answer texts corresponding to each question text; The third processing unit is used to determine multiple target texts related to the intent category from the full text based on the answer texts corresponding to each question text; The construction unit is used to construct a text library corresponding to the intent category based on the plurality of question texts, the answer texts corresponding to each question text, and the plurality of target texts; The third processing unit is configured to determine, based on the answer text corresponding to each question text, multiple target texts related to the intent category from the full text, including: Based on the answer text corresponding to each of the aforementioned question texts, determine the target center vector corresponding to the intent category; Clustering is performed on the entire text to obtain multiple clustering results; Determine the similarity between the target center vector and the center vectors corresponding to each of the clustering results; The full text included in the clustering result corresponding to the maximum similarity is determined as the multiple target texts.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for constructing a text library as described in any one of claims 1 to 5.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for constructing a text library as described in any one of claims 1 to 5.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for constructing a text library as described in any one of claims 1 to 5.