Label information determination method and apparatus, readable storage medium, and electronic device

Abstract

The application provides a label information determination method and device, a readable storage medium and an electronic device. The method comprises the following steps: determining a basic chapter unit corresponding to a text to be analyzed and a word in the basic chapter unit; for each word, determining word frequency information of the word, co-occurrence word information in the basic chapter unit, and first quantity information of the basic chapter unit in which the word appears; in each word, determining a high-frequency word that meets a preset high-frequency word condition in terms of the word frequency information and the first quantity information, and a secondary high-frequency word that meets a preset secondary high-frequency word condition; for each high-frequency word, determining a term set corresponding to the secondary high-frequency word, taking second quantity information of the basic chapter unit in which the term set appears as a support count, performing frequent term set mining, and determining a target frequent term set; and determining label information of the text to be analyzed based on the high-frequency word, co-occurrence word information corresponding to the high-frequency word, and the target frequent term set, so that the accuracy of the label information is higher.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and more particularly, to a label information determination method and device, a readable storage medium, and an electronic device. BACKGROUND

[0002] With the development of Internet technology, the amount of text data is increasing, and it is crucial to determine the label information corresponding to the text data, which can represent the core content of the text data. Currently, when determining the label information, the text data is subjected to cluster analysis, and the existing clustering algorithm cannot accurately express the text content when determining the label information, resulting in low accuracy of the obtained label information. SUMMARY

[0003] The present application provides a label information determination method, device, computer readable storage medium and electronic device to solve the technical problem of low accuracy of the label information determined in the prior art.

[0004] According to a first aspect of the present application, a label information determination method is provided, comprising:

[0005] determining a basic chapter unit corresponding to the text to be analyzed and a word item in the basic chapter unit;

[0006] for each word item, determining the word frequency information of the word item, the co-occurrence word information in the basic chapter unit, and the first number information of the basic chapter unit in which the word item appears;

[0007] among the various word items, determining a high-frequency word item that meets the preset high-frequency word condition in terms of the word frequency information and the first number information, and a secondary high-frequency word item that meets the preset secondary high-frequency word condition;

[0008] for each high-frequency word item, determining the item set corresponding to the secondary high-frequency word item, taking the second number information of the basic chapter unit in which the item set appears as the support count, performing frequent item set mining, and determining a target frequent item set;

[0009] based on the high-frequency word item, the co-occurrence word information corresponding to the high-frequency word item, and the target frequent item set, determining the label information of the text to be analyzed.

[0010] Optionally, the step of determining, for each high-frequency word item, the item set corresponding to the secondary high-frequency word item, taking the second number information of the basic chapter unit in which the item set appears as the support count, performing frequent item set mining, and determining a target frequent item set comprises:

[0011] For each high-frequency term, based on the second high-frequency term and the co-occurrence word information corresponding to the second high-frequency term, a term set corresponding to the second high-frequency term is determined, and second quantity information of a basic chapter unit in which the term set appears is taken as a support count;

[0012] In the term set of the second high-frequency term, a frequent term set with a support count greater than a support threshold is determined;

[0013] Based on a preset value, a target frequent term set is determined in the frequent term set.

[0014] Optionally, the determining of the target frequent term set in the frequent term set based on the preset value comprises:

[0015] In the frequent term set, a closed frequent term set containing a number of terms greater than or equal to 2 and less than or equal to a preset value minus one is determined;

[0016] The frequent term set containing a number of terms equal to the preset value and the closed frequent term set are taken as the target frequent term set.

[0017] Optionally, before the step of determining the basic chapter unit corresponding to the text to be analyzed and the terms in the basic chapter unit, the method further comprises:

[0018] The obtained original review text is cleaned and clustered to obtain a data cluster set;

[0019] In the data cluster set, a text to be analyzed is selected.

[0020] Optionally, the determining of the basic chapter unit corresponding to the text to be analyzed and the terms in the basic chapter unit comprises:

[0021] A stop word list suitable for the text to be analyzed is obtained;

[0022] Based on a preset segmentation rule and the stop word list, the text to be analyzed is processed to obtain the basic chapter unit corresponding to the text to be analyzed and the terms in the basic chapter unit.

[0023] Optionally, the determining of the term frequency information, the co-occurrence word information in the basic chapter unit, and the first quantity information of the basic chapter unit in which the term appears for each term comprises:

[0024] Based on the identification information of the basic chapter unit, a first dimension is determined;

[0025] Based on the terms in the basic chapter unit, a second dimension is determined;

[0026] construct a two-dimensional contingency table based on the first dimension and the second dimension;

[0027] fill the two-dimensional contingency table according to the target first dimension and the target second dimension corresponding to each word item, and determine a filled two-dimensional contingency table;

[0028] determine word frequency information, co-occurrence word information in the basic chapter unit, and first quantity information of basic chapter units in which the word item appears, based on the filled two-dimensional contingency table.

[0029] Optionally, the preset high-frequency word condition includes that the word frequency ranks in the first preset position, and the ratio of the first quantity information to the basic chapter unit quantity is in a first preset range.

[0030] The second high-frequency condition includes that the word frequency is greater than 2 and less than the word frequency ranking in the first preset position, and the first quantity information is in a second preset range.

[0031] According to a second aspect of the present application, a label information determination apparatus is provided, comprising:

[0032] a first determination module configured to determine a basic chapter unit corresponding to a text to be analyzed and word items in the basic chapter unit;

[0033] a second determination module configured to determine, for each word item, word frequency information of the word item, co-occurrence word information in the basic chapter unit, and first quantity information of basic chapter units in which the word item appears;

[0034] a third determination module configured to determine, among various word items, high-frequency word items whose word frequency information and first quantity information meet a preset high-frequency word condition, and second high-frequency word items meeting a preset second high-frequency word condition;

[0035] a fourth determination module configured to determine, for each high-frequency word item, an item set corresponding to the second high-frequency word item, take second quantity information of basic chapter units in which the item set appears as a support count, perform frequent item set mining, and determine a target frequent item set;

[0036] a fifth determination module configured to determine label information of the text to be analyzed based on the high-frequency word item, co-occurrence word information corresponding to the high-frequency word item, and the target frequent item set.

[0037] According to a third aspect of the present application, a computer readable storage medium is provided, the storage medium stores a computer program, and the computer program is used to execute the above label information determination method.

[0038] According to a fourth aspect of the present application, an electronic device is provided, comprising:

[0039] a processor;

[0040] a memory for storing the executable instructions of the processor;

[0041] the processor is configured to read the executable instructions from the memory and execute the instructions to implement the above-mentioned label information determination method.

[0042] Compared with the prior art, the label information determination method, device, computer readable storage medium and electronic equipment provided by the present application have at least the following beneficial effects:

[0043] The technical solution of the present application first determines the basic chapter unit corresponding to the text to be analyzed and the word items in the basic chapter unit, then counts the relevant information of the word items in the basic chapter unit, determines the word frequency information of each word item, the co-occurrence word information of each word item in the basic chapter unit, and the first quantity information of the basic chapter unit in which each word item appears. The high-frequency word condition and the second high-frequency word condition related to the word frequency information and the first quantity information are set in advance, so that after counting the relevant information of the word items, the high-frequency word items that meet the preset high-frequency word condition and the second high-frequency word items that meet the preset second high-frequency word condition can be selected from the word items. For each determined high-frequency word item, the item set corresponding to the second high-frequency word item is determined, and the second quantity information of the basic chapter unit in which the item set appears is used as the support count for frequent item set mining, and the target frequent item set is determined. After the second high-frequency word item is mined for the frequent item set, the label information of the text to be analyzed is further determined according to the determined high-frequency word item, the co-occurrence word information corresponding to the high-frequency word item, and the mined target frequent item set. In the technical solution provided by the present application, the basic chapter unit is used as the analysis unit to realize the refinement of the analysis unit, and the high-frequency word item, the co-occurrence word information corresponding to the high-frequency word item, and the target frequent item set composed of the second high-frequency word item accurately reflect the obvious features of the text to be analyzed, thereby making the accuracy of the determined label information higher and meeting the requirements of fine-grained analysis. Moreover, by mining only the second high-frequency word item for the frequent item set, the number of word items used for frequent item set mining is effectively reduced, the computing cost is reduced, and the label information determination efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the description of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative labor.

[0045] Figure 1 is a flowchart of the label information determination method provided by an exemplary embodiment of the present application Figure 1;

[0046] Figure 2 is a flowchart of a label information determination method provided by an exemplary embodiment of the present application Figure 2 ;

[0047] Figure 3 is a structural diagram of a label information determination device provided by an exemplary embodiment of the present application

[0048] Figure 4 is a structural diagram of an electronic device provided by an exemplary embodiment of the present application DETAILED DESCRIPTION

[0049] The technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0050] Exemplary method

[0051] Figure 1 is a flowchart of a label information determination method provided by an exemplary embodiment of the present application, and at least includes the following steps:

[0052] Step 11, determining an elementary discourse unit corresponding to the text to be analyzed and a word item in the elementary discourse unit.

[0053] The text to be analyzed is text data that needs to be analyzed. Specifically, the text to be analyzed is user comment text obtained through a normal channel, which is non-structured data. The user comment text includes but is not limited to public comments on social software, complaints in emails or telephone, etc. The complaints in emails or telephone are internal data of the user using the method provided by the present embodiment. Of course, the text to be analyzed can also be data other than user comment text, such as text data related to a target field, for example, commodity text data, recommended text data, etc.

[0054] An elementary discourse unit (EDU) is a basic analysis unit in discourse linguistics, which divides a discourse into several units. In this step, the text to be analyzed is segmented into different elementary discourse units, which realizes the refinement of the analysis unit and is conducive to obtaining more accurate label information subsequently.

[0055] A word item refers to a single word obtained after word segmentation of the text to be analyzed. Word segmentation refers to dividing Chinese character sequences in the text to be analyzed into single words. Word segmentation methods include, but are not limited to, string matching word segmentation methods, semantic word segmentation methods, and statistical word segmentation methods. The string matching word segmentation methods include forward maximum matching, backward maximum matching, and shortest path word segmentation.

[0056] Specifically, the text to be analyzed is segmented to determine the word items corresponding to the text to be analyzed, and then the preselected EDU segmentation rule is used to segment the text to be analyzed to obtain the basic discourse units and the word items in the basic discourse units corresponding to the text to be analyzed. Of course, the preselected EDU segmentation rule can also be used to segment the text to be analyzed to obtain the basic discourse units, and then the basic discourse units are further segmented to obtain the word items in each basic discourse unit.

[0057] For example, the obtained text to be analyzed is: The park of XX property management is not clean. After rainy and snowy weather, the road is slippery and not dehumidified. The cleaning is not thorough. The preselected EDU segmentation rule is determined. Different EDU segmentation rules correspond to different segmentation granularities and different segmentation results. If a period is selected as the segmentation rule, the obtained basic discourse units and the word items in the basic discourse units can be [The park of XX property management is not clean.] as EDU1, [After rainy and snowy weather, the road is slippery and not dehumidified.] as EDU2, and [The cleaning is not thorough.] as EDU3. If a comma and a period are selected as the segmentation rule, [The park of XX property management is not clean.] is EDU1, [After rainy and snowy weather,] is EDU2, [The road is slippery,] is EDU3, [and not dehumidified.] is EDU4, and [The cleaning is not thorough] is EDU5. Of course, a paragraph marker such as / n / r can also be selected as the segmentation rule, that is, one paragraph is one EDU. It should be noted that in actual application scenarios, the user can preselect the segmentation rule according to actual needs. The specific content of the segmentation rule is not limited in this embodiment.

[0058] In some embodiments, before step 11, the method further comprises:

[0059] Step 10: performing cleaning and clustering processing on the obtained original review text to obtain a data clustering set; in the data clustering set, a text to be analyzed is selected.

[0060] The original review text is the user review data obtained without processing. If the user comments through voice, the user voice data obtained is subjected to voice recognition to determine the comment text. The comment texts obtained in various ways are summarized to obtain the original review text.

[0061] Specifically, as Figure 2As shown, the obtained original review text is taken as the corpus to be analyzed, the corpus cleaning module is called to clean the corpus, and then the topic clustering model is called to preliminarily classify the cleaned original review text to obtain a data clustering set. Each data clustering set corresponds to a topic, and some topics do not need to be subjected to subsequent steps, so selection is performed on the obtained data clustering set, and the category to be analyzed, i.e., the text to be analyzed, is selected. That is, the text to be analyzed is part of the data clustering set, so the selection after preliminary classification effectively reduces the amount of data subjected to subsequent steps and improves the calculation efficiency.

[0062] In a possible implementation, a screening standard of the clustering set is preset, and the text to be analyzed is selected from the obtained data clustering set according to the screening standard. In another possible implementation, after the data clustering set is obtained, the text to be analyzed is selected from the data clustering set in response to a selection instruction of a user, that is, the user intervenes in the selection of data, and the text to be analyzed subjected to subsequent steps is selected.

[0063] In some embodiments, the step 11 comprises:

[0064] The step 111 comprises obtaining a stop word table suitable for the text to be analyzed.

[0065] The stop word table is used to record automatically filtered words or terms. Specifically, the text to be analyzed can correspond to different characteristics, and therefore, the stop word table suitable for the text to be analyzed needs to be obtained, that is, the general stop word table is adjusted according to the characteristics of the text to be analyzed. Specifically, in the general stop word table, function words are removed and only content words are left, but when the text to be analyzed is user review text, i.e., user feedback analysis is performed, some function words, such as negative adverbs, play an important role in understanding user intent, and some content words, such as content words appearing in each EDU, can be unimportant, so the general stop word table is adjusted to obtain the stop word table suitable for the text to be analyzed.

[0066] The step 112 comprises processing the text to be analyzed based on a preset segmentation rule and the stop word table to obtain a basic discourse unit corresponding to the text to be analyzed and terms in the basic discourse unit.

[0067] Specifically, the EDU segmentation rule is preset, the text to be analyzed is segmented, the basic discourse unit corresponding to the text to be analyzed is obtained, and the basic discourse unit corresponding to the text to be analyzed is subjected to word segmentation and stop word table processing, so as to obtain the basic discourse unit corresponding to the text to be analyzed and the terms in the basic discourse unit.

[0068] In a possible implementation, the clustering analysis module and the word segmentation module are preset, the clustering analysis model is used to call a classification manner based on a clustering topic model (Latent Dirichlet Allocation, LDA) and a K-means clustering algorithm to perform clustering processing to obtain a data clustering set according to a text situation, and the classification data granularity obtained in the data clustering set is usually difficult to meet the requirement of fine-grained analysis. Therefore, the to-be-analyzed text that needs to be analyzed is selected from the data clustering set, the to-be-analyzed text is segmented based on a preset segmentation rule to obtain an EDU, and the segmented result is re-segmented to remove stop words, that is, when it is detected that any one word after word segmentation is the same as a word in a stop word table, the same word is deleted, and then the basic discourse unit corresponding to the to-be-analyzed text and the word in the basic discourse unit are obtained.

[0069] In step 12, for each word, word frequency information, co-occurrence word information in the basic discourse unit, and first quantity information of the basic discourse unit in which the word appears are determined.

[0070] The word frequency information is used to represent the frequency of the word. According to the Zipf distribution, in a natural language corpus, the frequency of a word is inversely proportional to its ranking in the frequency table. The frequency of a high-frequency word is often several times that of a low-frequency word. The frequency of a low-frequency word is low, but the quantity is much higher than that of a high-frequency word.

[0071] The co-occurrence word is a word that often collocates and co-occurs in text data. In this step, for any word, the co-occurrence word of the word is a word cluster that appears in the same EDU as the word. The co-occurrence word information includes the co-occurrence word of the word and the co-occurrence times.

[0072] For any word, the basic discourse unit in which the word appears can be a target basic discourse unit, and the first quantity information of the target basic discourse unit is a statistical value of the number of basic discourse units in which the word appears.

[0073] Specifically, after the basic discourse unit and the word in the basic discourse unit are obtained, the word-related information in the basic discourse unit is counted, and the word frequency information, the co-occurrence word information, and the first quantity information of the basic discourse unit in which the word appears are counted.

[0074] In some embodiments, the step 12 includes:

[0075] In step 121, a first dimension is determined based on the identification information of the basic discourse unit.

[0076] The identification information of the basic discourse unit (ID of the basic discourse unit) has uniqueness and is used to distinguish the basic discourse units, such as EDU1, EDU2, EDU3, and the like. The identification information of the basic discourse unit is taken as the first dimension.

[0077] In step 122, a second dimension is determined based on the word items in the basic discourse unit.

[0078] The second dimension is a word item dimension, and specifically, after the repeated word items in all word items are processed by merging, one unique word item is obtained, and the second dimension is obtained. For example, [After a rainy and snowy weather, the road surface is wet and slippery, and the humidity is not removed.][The cleaning is not thorough.] both contain the word item “not”, so the repeated word item “not” is merged to obtain one unique word item “not”, and the obtained second dimension can be [rain, snow, weather, after, road surface, wet and slippery, not, remove humidity, clean, thoroughly].

[0079] In step 123, a two-dimensional contingency table is constructed based on the first dimension and the second dimension.

[0080] Specifically, the first dimension is taken as the row marker, and the second dimension is taken as the column marker to construct the two-dimensional contingency table, and the content in the two-dimensional contingency table is a preset initial value, such as 0. Alternatively, the second dimension is taken as the row marker, and the first dimension is taken as the column marker to construct the two-dimensional contingency table, and the content in the two-dimensional contingency table is a preset initial value, such as 0.

[0081] In step 124, the two-dimensional contingency table is filled for each target first dimension and target second dimension corresponding to a word item, and a filled two-dimensional contingency table is determined.

[0082] Specifically, each word item in the basic discourse unit has a corresponding target first dimension and target second dimension, and the target first dimension and target second dimension have a corresponding target position in the two-dimensional contingency table. Therefore, the value at the target position of the word item in the two-dimensional contingency table is incremented by one to fill the two-dimensional contingency table, and the filled two-dimensional contingency table is determined.

[0083] For example, the first dimension is EDU1, EDU2, EDU3, and the like, and the second dimension is [XX, property management], and the initial value of the two-dimensional contingency table is set to 0. For the word item XX, the target first dimension corresponding to XX is EDU1, and the target second dimension is XX. If XX appears once in EDU1, the 0 at the target position of the two-dimensional contingency table with the first dimension EDU1 and the second dimension XX is rewritten as 1. If XX appears twice in EDU1, the 0 at the target position of the two-dimensional contingency table with the first dimension EDU1 and the second dimension XX is rewritten as 2. That is, a two-dimensional contingency table of i*j is constructed, and a word item j appears n times in the i-th row, that is, n is added to the i-th row and the j-th column.

[0084] Step 125, based on the filled two-dimensional contingency table, determine the word frequency information of each word item, the co-occurrence word information in the basic chapter unit, and the first number information of the basic chapter unit in which the word item appears.

[0085] Specifically, in the filled two-dimensional contingency table, the word frequency information, the co-occurrence word information, and the first number information of the basic chapter unit in which the word item appears can be determined for each word item belonging to the basic chapter unit and other word items in the same basic chapter unit. For example, there are word items "XX" and "rain", etc. For the word item "XX", the word frequency information of "XX", the co-occurrence word information of "XX", and the first number information of the basic chapter unit in which "XX" appears are determined in EDU1-EDU4. For the word item "rain", the word frequency information of "rain", the co-occurrence word information of "rain", and the first number information of the basic chapter unit in which "rain" appears are determined in EDU1-EDU4.

[0086] Step 13, among the various word items, determine the high-frequency word items that meet the preset high-frequency word condition of the word frequency information and the first number information, and the secondary high-frequency word items that meet the preset secondary high-frequency word condition.

[0087] In this step, the high-frequency word condition and the secondary high-frequency word condition related to the word frequency information and the first number information are set in advance. Meeting the preset high-frequency word condition indicates that the word item is a high-frequency word item, and meeting the preset secondary high-frequency word condition indicates that the word item is a secondary high-frequency word item.

[0088] Specifically, according to the preset high-frequency word condition, it is judged whether the word item is a high-frequency word. If yes, the high-frequency word is output, and if not, according to the preset secondary high-frequency word condition, it is judged whether the word item is a secondary high-frequency word item. If yes, step 14 is executed, and if not, the word item is not processed.

[0089] In some embodiments, the preset high-frequency word condition includes that the word frequency ranking is in the front preset bit, and the ratio of the first number information to the number of basic chapter units is in the first preset range. The secondary high-frequency word condition includes that the word frequency is greater than 2 and less than the word frequency ranking in the front preset bit, and the first number information is in the second preset range.

[0090] Specifically, the word frequency information is a specific numerical value, and the word frequency information of all word items is sorted from large to small, so the more times a word item appears, the higher the word frequency ranking, and the more important the word item. Furthermore, the high-frequency word condition and the next high-frequency word condition are defined according to the word frequency ranking of each word item and the first quantity information. For example, the word items with a word frequency ranking in the top X and a ratio of the first quantity information to the number of basic chapter units in a first preset range are determined as high-frequency word items, where the ratio of the first quantity information to the number of basic chapters is the probability of the word item appearing in the EDU, and the first preset range can be 50%-100%. According to Zipf's law, words with a word frequency greater than 2 are often the dividing line of the long tail, and words appearing in more than 2-4 EDUs often indicate a typified event, so the next high-frequency word condition is defined as a word frequency greater than 2 but less than the word frequency of a high-frequency word, i.e., less than the word frequency of a word with a word frequency ranking in the top preset position, and appearing in at least Y EDUs, where Y is a second preset range, and Y can be greater than or equal to 2 and less than or equal to 4.

[0091] As shown in Figure 2 , it is judged whether the word frequency information and the first quantity information of the word item meet the high-frequency word condition, and if so, the high-frequency word frequency, the high-frequency word co-occurrence word, and the co-occurrence times are output. If not, it is further judged whether the word item meets the next high-frequency condition, and if not, no processing is performed, and if so, step 14 is performed.

[0092] In step 14, for each high-frequency word item, the item set corresponding to the next high-frequency word item is determined, the second quantity information of the basic chapter unit in which the item set appears is taken as the support count, frequent item set mining is performed, and the target frequent item set is determined.

[0093] Wherein, the item set refers to a collection of several items, and an item set containing K word items is referred to as a K-item set. The support count refers to the frequency of a certain set appearing in all transactions. In this step, the second quantity information is taken as the support count, that is, in this step, attention is paid to whether the item set corresponding to the next high-frequency word item appears in the basic chapter unit, and the number of times the item set appears in a basic chapter unit is not concerned. Taking the second quantity information as the support count can more objectively reflect the distribution of the next high-frequency word item in the EDU. The frequent item set refers to a set with a support count greater than or equal to a support threshold (minimum support). The frequent item set can indicate variables that often appear together in the basic chapter unit, which is helpful for building context and plays an important role in finally determining the label information, so the next high-frequency word item is subjected to frequent item set mining to determine the target frequent item set.

[0094] Optionally, the target frequent item set is all frequent item sets mined from the next high-frequency word item.

[0095] Optionally, the target frequent item set is a closed frequent item set of all frequent item sets mined after mining the sub-high frequency word item. The closed frequent item set refers to a frequent item set whose direct superset does not have the same support count as it.

[0096] For example, there are EDU1 "No pruning branches are cleaned up, the cart makes the community messy after passing by, and the cleaning and sanitation need to be strengthened", EDU2 "Green pruning needs to be cleaned up in time, pay so many property fees, and hope to at least do a good job in sanitation, which will affect the house price", EDU3 "The stairs and corridors are very dirty, please arrange someone to handle it now", and EDU4 "There are dead branches and leaves everywhere, the property prunes flowers and plants, but does not clean up in time, even the sanitation is not clean, and it says five-star home". After performing word segmentation and stop words on EDU1-EDU4, an exemplary two-dimensional contingency table is as follows:

[0097]

[0098]

[0099] Among them, the sub-high frequency word item "clean up" appears in EDU1, EDU2 and EDU4, so the word items co-occurring with the sub-high frequency word item "clean up" in EDU1, EDU2 and EDU4 and the co-occurrence frequency (not counted once, such as EDU3) are counted to calculate the item set of the sub-high frequency word item "clean up".

[0100] Specifically, five word items [clean, clean up, sanitation, pruning, in time] are obtained:

[0101] 5 choose 2, excluding those without "clean up", there are four 2-item sets: [clean, clean up] [sanitation, clean up] [pruning, clean up] [in time, clean up];

[0102] 5 choose 3, excluding those without "clean up", there are six 3-item sets: [clean, sanitation, clean up] [clean, clean up, pruning] [clean, clean up, in time] [clean up, sanitation, pruning] [clean up, sanitation, in time] [clean up, pruning, in time];

[0103] 5 choose 4, excluding those without "clean up", there are four 4-item sets: [clean, clean up, sanitation, pruning] [clean up, sanitation, pruning, in time] [clean, clean up, sanitation, in time] [clean, clean up, pruning, in time];

[0104] 5 choose 5, excluding those without "clean up", there is one 5-item set: [clean, clean up, sanitation, pruning, in time]

[0105] Among them, the item set with a support count of 3 / 4 is:

[0106] [clean, clean][hygienic, clean][trimmed, clean][clean, hygienic, clean][trimmed, clean, clean][trimmed, hygienic, clean][trimmed, hygienic, clean];

[0107] The itemsets with a support count of 2 / 4 are:

[0108] [Timely, clean][Timely, clean, clean][Timely, hygienic, clean][Timely, trim, clean][Trimming, hygienic, timely, clean][Clean, hygienic, timely, clean][Clean, clean, trimming, timely][Hygienic, clean, trimming, timely];

[0109] Now, let's set the support threshold to 3 / 4, meaning we need to find the itemset consisting of combinations of terms that appear simultaneously with "cleaning" in at least 3 EDU entries. By definition, this itemset set consists of 2-itemsets, 3-itemsets, and 4-itemsets: {[pruning, cleaning][hygiene, cleaning][clean, cleaning][pruning, hygiene, cleaning][pruning, clean, cleaning][clean, hygiene, cleaning][pruning, hygiene, clean, cleaning]}, where the closed frequent itemset is "[hygiene, clean, cleaning, pruning]"; if we set the support threshold to 2 / 4, then the itemset set is: {[ The set of closed frequent itemsets is “[hygiene, clean, clean, trim]” and “[hygiene, clean, trim, timely]”.

[0110] Optionally, itemset selection rules can be pre-set. After mining the second most frequent terms, the itemset selection rules can be used to select from all the mined frequent itemsets to select the target frequent itemset. Users can set the itemset selection rules according to the actual application scenario.

[0111] In some embodiments, step 14 includes:

[0112] Step 141: For each high-frequency term, based on the second-highest frequency term and the co-occurrence information corresponding to the second-highest frequency term, determine the itemset corresponding to the second-highest frequency term, and use the second quantity information of the basic discourse units in which the itemset appears as the support count.

[0113] Specifically, for any one sub-high-frequency term, the co-occurrence word information corresponding to the sub-high-frequency term is determined, i.e., the co-occurrence term and the co-occurrence frequency of the sub-high-frequency term, and various combinations of the sub-high-frequency term and the co-occurrence word are taken as the item set set corresponding to the sub-high-frequency term, the item set set is composed of various item sets, and an exemplary item set set is {[prune, clean up] [hygiene, clean up] [clean, clean up] [prune, hygiene, clean up] [prune, clean, clean up] [clean, hygiene, clean up] [prune, hygiene, clean, clean up]}, wherein [prune, clean up] and the like are item sets, so that multiple item sets corresponding to the sub-high-frequency term are often determined. For each item set, the second quantity information of the basic chapter unit in which the item set appears is determined, and the second quantity information is taken as the support count, that is, all item sets co-occurring with the sub-high-frequency term, regardless of the internal order of the item set, appear in how many EDUs, which is the basis for the support count. For example, the second quantity information of the basic chapter unit in which [prune, clean up] appears is determined, the second quantity information of the basic chapter unit in which [hygiene, clean up] appears is determined, and so on. If the 2-item set [prune, clean up] appears in 3 of 4 EDUs, the support of [prune, clean up] is 3 / 4.

[0114] Step 142, in the item set of the sub-high-frequency term, a frequent item set with a support count greater than a support threshold is determined.

[0115] Step 143, based on a preset value, a target frequent item set is determined in the frequent item set.

[0116] Specifically, the second quantity information is taken as the support count, and in the item set of the sub-high-frequency term, a frequent item set with a support count greater than a support count threshold is determined. A preset value is set in advance, and the preset value is used to truncate the number of terms in the item set. For example, the item set of the sub-high-frequency term includes 20 item sets, and the preset value 8 is used to truncate 20, so that when the target frequent item set is determined, the frequent item set or the closed frequent item set containing the number of terms greater than the preset value does not need to be considered. Thus, by the preset value, the amount of operation data is effectively reduced, and the operation speed is improved.

[0117] Optionally, the frequent item set containing the number of terms less than or equal to the preset value is selected as the target frequent item set in the frequent item set.

[0118] Optionally, the closed frequent item set containing the number of word items less than or equal to the preset value is selected from the frequent item set as the target frequent item set. For example, in the above example, the closed frequent item set determined is "[hygiene, clean, clean, pruning] and [hygiene, clean, clean, pruning, and timely]", if there is a preset value of 4, i.e. the number of items of the closed frequent item set is truncated by 4, i.e. the closed frequent item set "[hygiene, clean, clean, pruning, and timely]" does not need to be determined, and the target frequent item set is determined as "[hygiene, clean, clean, pruning]".

[0119] In some embodiments, the step 143 comprises:

[0120] In the frequent item set, the closed frequent item set containing the number of word items greater than or equal to 2 and less than or equal to a preset value minus one is determined;

[0121] The frequent item set containing the number of word items equal to the preset value and the closed frequent item set are taken as the target frequent item set.

[0122] In this embodiment, the closed frequent item set is determined in the frequent item set of the second high frequency word item, but because finding all the closed frequent item set has a large time overhead, the preset value is truncated, i.e. the closed frequent item set greater than or equal to 2 and less than or equal to a preset value minus one is found, for example, the preset value is m, then 2~m-1 closed frequent item sets are determined, and m frequent item sets are determined, and then 2~m-1 closed frequent item sets and m frequent item sets are determined as the target frequent item set.

[0123] Specifically, the basic chapter unit is the result obtained after the text features reflecting the event / phenomenon are segmented, and the key word items required to depict a label information are between 1~8, so the preset value can be greater than or equal to 1 and less than or equal to 8, further, the preset value can be greater than or equal to 3 and less than or equal to 8. On the other hand, when the dimension of the word table of the second high frequency word item is large, it may affect the mining efficiency of the algorithm, therefore we can limit k to 8 item sets according to the linguistic features, so as to effectively reduce the amount of operation data and improve the operation speed by the preset value.

[0124] Step 15, based on the high frequency word item, the co-occurrence word information corresponding to the high frequency word item, and the target frequent item set, the label information of the text to be analyzed is determined.

[0125] In an embodiment, the determined high-frequency word item and the co-occurrence word information corresponding to the high-frequency word item can construct a strong context effect, and thus after the word item is determined as a high-frequency word item, the high-frequency word item can be directly utilized in step 15, without further mining of the frequent item set for the high-frequency word item, effectively reducing the amount of data for mining of the frequent item set, saving computing overhead, and improving data computing speed. Meanwhile, the mining of the frequent item set for the sub-high-frequency word item is performed to determine the target frequent item set, and the determined target frequent item set can also reflect the text features, and thus the high-frequency word item, the co-occurrence word information of the high-frequency word item, and the target frequent item set that can reflect the text features are comprehensively considered, so that the accuracy of the determined label information of the text to be analyzed is relatively high.

[0126] Specifically, the high-frequency word item, the co-occurrence word information corresponding to the high-frequency word item, and the target frequent item set are displayed, and the label information of the text to be analyzed is determined in response to an input instruction of a user. That is, the high-frequency word item, the co-occurrence word information, and the target frequent item set are provided for a business expert to summarize, and the business expert inputs the label information of the text to be analyzed after summarizing.

[0127] In a possible application scenario, the text to be analyzed is a user comment text, and the label information is a Voice of Customer (VoC) label. The VoC label can quickly understand user demand and understand the perception of users on products or services. In particular, the VoC label has a relatively high requirement on analysis granularity, and is usually analyzed in a fine-grained manner. Therefore, it is difficult to obtain a fine-grained VoC label through simple clustering analysis. Figure 2 In the embodiment shown in FIG. 6, the text to be analyzed is preliminarily analyzed through clustering analysis, and a basic EDU is taken as an analysis unit, so that the EDU feature is introduced, the high-frequency word condition and the sub-high-frequency word condition are further defined by using the Zipf distribution feature of words, the Zipf distribution feature is introduced, the Zipf distribution feature of words and the EDU feature are taken as constraint strategies, and are introduced into the frequent item set mining algorithm process, so that the analysis and calculation range is reasonably reduced, the obvious features are quickly mined, and the label information with relatively high accuracy and fine granularity is constructed.

[0128] In the above embodiment, the basic chapter unit corresponding to the text to be analyzed and the word items in the basic chapter unit are determined first, and then the relevant information of the word items in the basic chapter unit is counted to determine the word frequency information of each word item, the co-occurrence word information of each word item in the basic chapter unit, and the first quantity information of the basic chapter unit in which each word item appears. The high-frequency word condition and the secondary high-frequency word condition related to the word frequency information and the first quantity information are set in advance, so that after counting the relevant information of the word items, the high-frequency word items whose word frequency information and first quantity information meet the preset high-frequency word condition and the secondary high-frequency word items whose word frequency information and first quantity information meet the preset secondary high-frequency word condition can be selected from the word items. For each determined high-frequency word item, the item set corresponding to the secondary high-frequency word item is determined, and the second quantity information of the basic chapter unit in which the item set appears is used as the support count for the frequent item set mining, and the target frequent item set is determined. After the frequent item set mining of the secondary high-frequency word item, the label information of the text to be analyzed is determined according to the determined high-frequency word item, the co-occurrence word information corresponding to the high-frequency word item, and the target frequent item set mined. In the technical solution provided in this embodiment, the basic chapter unit is used as the analysis unit to realize the refinement of the analysis unit, and the high-frequency word item, the co-occurrence word information corresponding to the high-frequency word item, and the target frequent item set composed of the secondary high-frequency word item accurately reflect the obvious features of the text to be analyzed, so that the accuracy of the determined label information is higher and meets the requirement of fine-grained analysis. Moreover, by only performing the frequent item set mining on the secondary high-frequency word item, the number of word items used for the frequent item set mining is effectively reduced, the calculation cost is reduced, and the label information determination efficiency is improved.

[0129] Exemplary apparatus

[0130] Based on the same concept as the method embodiment of the present application, the label information determination device is also provided in the embodiment of the present application.

[0131] Figure 3 The structure schematic diagram of the label information determination device provided in an exemplary embodiment of the present application is shown, which comprises:

[0132] The first determination module 31 is configured to determine the basic chapter unit corresponding to the text to be analyzed and the word items in the basic chapter unit.

[0133] The second determination module 32 is configured to determine, for each word item, the word frequency information of the word item, the co-occurrence word information in the basic chapter unit, and the first quantity information of the basic chapter unit in which the word item appears.

[0134] The third determination module 33 is configured to determine, from the word items, the high-frequency word items whose word frequency information and first quantity information meet the preset high-frequency word condition, and the secondary high-frequency word items whose word frequency information and first quantity information meet the preset secondary high-frequency word condition.

[0135] The fourth determination module 34 is configured to determine, for each high-frequency term, a term set corresponding to the second high-frequency term, take second quantity information of basic chapter units in which the term set appears as a support count, perform frequent term set mining, and determine a target frequent term set.

[0136] The fifth determination module 35 is configured to determine label information of the text to be analyzed based on the high-frequency term, the co-occurrence word information corresponding to the high-frequency term, and the target frequent term set.

[0137] In some embodiments, the fourth determination module includes:

[0138] The data determination unit is configured to determine, for each high-frequency term, a term set of the second high-frequency term based on the second high-frequency term and the co-occurrence word information corresponding to the second high-frequency term, and take second quantity information of basic chapter units in which the term set appears as a support count.

[0139] The data mining unit is configured to determine, in the term set of the second high-frequency term, a frequent term set in which the support count is greater than a support threshold.

[0140] The target determination unit is configured to determine, in the frequent term set, a target frequent term set based on a preset value.

[0141] In some embodiments, the target determination unit is specifically configured to determine, in the frequent term set, a closed frequent term set containing a number of terms greater than or equal to 2 and less than or equal to a preset value minus 1, and take the closed frequent term set and a frequent term set containing a number of terms equal to the preset value as the target frequent term set.

[0142] In some embodiments, the apparatus further includes:

[0143] The text acquisition module is configured to perform cleaning processing and clustering processing on the obtained original review text to obtain a data clustering set, and select a text to be analyzed in the data clustering set.

[0144] In some embodiments, the first determination module includes:

[0145] The vocabulary acquisition unit is configured to acquire a stop word table suitable for the text to be analyzed.

[0146] The text processing unit is configured to process the text to be analyzed based on a preset segmentation rule and the stop word table to obtain basic chapter units corresponding to the text to be analyzed and terms in the basic chapter units.

[0147] In some embodiments, the second determination module includes:

[0148] The first determining unit is used to determine the first dimension based on the identification information of the basic chapter unit;

[0149] The second determining unit is used to determine the second dimension based on the terms in the basic text unit;

[0150] The contingency table construction unit is used to construct a two-dimensional contingency table based on the first dimension and the second dimension;

[0151] The contingency table filling unit is used to fill the two-dimensional contingency table for each term's target first dimension and target second dimension, and to determine the filled two-dimensional contingency table.

[0152] The third determining unit is used to determine the word frequency information of each term, the co-occurring word information in the basic discourse unit, and the first quantity information of the basic discourse unit in which the term appears, based on the filled two-dimensional contingency table.

[0153] In some embodiments, the preset high-frequency word condition includes word frequency ranking in the top preset position, and the ratio of the first quantity information to the number of basic text units is within a first preset range;

[0154] The second highest frequency condition includes a word frequency greater than 2 and less than the word frequency of the first preset position, and the first quantity information is within a second preset range.

[0155] Exemplary electronic device

[0156] Figure 4 A block diagram of an electronic device according to an embodiment of the present invention is shown.

[0157] like Figure 4 As shown, the electronic device 40 includes one or more processors 41 and a memory 42.

[0158] The processor 41 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 40 to perform desired functions.

[0159] The memory 42 can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read only memory (ROM), hard disk, flash memory, and / or the like. The computer-readable storage media can store one or more computer program instructions executable by the processor 41 to implement the tag information determination method of the embodiments of the present application described above and / or other desired functions.

[0160] In one example, the electronic device 40 can further include an input device 43 and an output device 44, which are interconnected through a bus system and / or other form of connection mechanism (not shown).

[0161] Of course, in order to simplify, Figure 4 Only some of the components of the electronic device 40 related to the present application are shown in FIG. 4, and components such as buses, input / output interfaces, and the like are omitted. In addition, the electronic device 40 can include any other appropriate components according to the specific application.

[0162] Exemplary computer program product and computer readable storage medium

[0163] In addition to the above-mentioned methods and devices, embodiments of the present application can also be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the tag information determination method according to various embodiments of the present application described in the above "Exemplary Methods" section of the specification.

[0164] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, and / or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote cloud device or server.

[0165] In addition, embodiments of the present application can also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the steps of the tag information determination method according to various embodiments of the present application described in the above "Exemplary Methods" section of the specification.

[0166] The computer readable storage medium can be a combination of one or more computer readable media. The computer readable media can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] The above description of the disclosed aspects merely exemplifies the general principles of the application. It is appreciated that the advantages, scope, and nature of the disclosure can be evident by those skilled in the art from the foregoing detailed description, and the examples as given are intended merely for purposes of illustration. Further, the description as set forth is not intended to be all-inclusive of the aspects of the application. Still further, the aspects of the application have been described with the intention of being illustrative, and not limiting, of the principles thereof. It is therefore contemplated that various modifications of the aspects will occur to those skilled in the art, and all such modifications are believed to be within the scope of the application as set forth in the appended claims.

[0168] The block diagrams of the devices, apparatuses, systems, and methods involved in the present application are merely illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be as shown in the block diagrams. As will be recognized by those skilled in the art, the devices, apparatuses, systems, and methods can be connected, arranged, and configured in any manner. Words such as "including," "containing," "comprising," and the like are to be construed in an inclusive fashion, meaning that "comprising" means "including, but not limited to." The words "coupled" and "coupling" are intended to mean either an indirect or direct electrical connection in accordance with the uses in context. The use of "including," "comprising," "having," "containing," "carrying" or any other variation thereof herein is not a limitation to the aspects of the application. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0169] It is also to be noted that the various components or steps can be combined, divided, re-arranged, omitted, eliminated, and / or re-arranged, in the devices, apparatuses, and methods of the present application. Such combinations, divisions, re-arrangements, omissions, eliminations, and / or re-arrangements are to be construed as equivalent variations of the present application.

[0170] The above description of the disclosed aspects merely exemplifies the general principles of the application. It is appreciated that the advantages, scope, and nature of the disclosure can be evident by those skilled in the art from the foregoing detailed description, and the examples as given are intended merely for purposes of illustration. Further, the description as set forth is not intended to be all-inclusive of the aspects of the application. Still further, the aspects of the application have been described with the intention of being illustrative, and not limiting, of the principles thereof. It is therefore contemplated that various modifications of the aspects will occur to those skilled in the art, and all such modifications are believed to be within the scope of the application as set forth in the appended claims.

[0171] The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the application to forms disclosed herein. Although several example aspects and embodiments have been discussed, those skilled in the art will recognize certain variations, modifications, changes, additions, and sub-combinations thereof.

Claims

1. A method for determining tag information, characterized in that, include: Determine the basic discourse unit corresponding to the text to be analyzed and the terms in the basic discourse unit; For each term, determine the term frequency information, the co-occurring word information in the basic discourse unit, and the first quantity information of the basic discourse units in which the term appears; Among each term, high-frequency terms that meet the preset high-frequency term conditions and the first quantity information are identified, as well as second-highest frequency terms that meet the preset second-highest frequency term conditions. For each high-frequency term, the itemset corresponding to the second highest-frequency term is determined. The second quantity information of the basic text units in which the itemset appears is used as the support count to perform frequent itemset mining and determine the target frequent itemset. Based on the high-frequency terms, the co-occurrence information corresponding to the high-frequency terms, and the target frequent itemset, the tag information of the text to be analyzed is determined.

2. The method according to claim 1, characterized in that, For each high-frequency term, the next highest-frequency term's corresponding itemset is determined. The second quantity of basic textual units containing the itemset is used as the support count to perform frequent itemset mining, identifying the target frequent itemset, including: For each high-frequency term, based on the second-highest frequency term and the co-occurrence word information corresponding to the second-highest frequency term, the itemset corresponding to the second-highest frequency term is determined, and the second quantity information of the basic discourse units in which the itemset appears is used as the support count; In the item set of the second most frequent terms, the frequent item set whose support count is greater than the support threshold is identified; Based on preset values, a target frequent itemset is determined from the frequent itemset.

3. The method according to claim 2, characterized in that, The step of determining the target frequent itemset based on preset values ​​in the frequent itemset includes: A closed frequent itemset is determined from the frequent itemset, wherein the number of terms contained in the closed frequent itemset is greater than or equal to 2 and less than or equal to a preset value minus one. The frequent itemsets containing the number of terms equal to the preset value, and the closed frequent itemsets, are taken as the target frequent itemsets.

4. The method according to claim 1, characterized in that, Before the step of determining the basic discourse unit corresponding to the text to be analyzed and the terms in the basic discourse unit, the method further includes: The obtained raw comment text is cleaned and clustered to obtain a data cluster set; The text to be analyzed is selected from the data cluster set.

5. The method according to claim 1, characterized in that, The process of determining the basic discourse unit corresponding to the text to be analyzed and the terms in the basic discourse unit includes: Obtain a stop word list that is appropriate for the text to be analyzed; The text to be analyzed is processed based on the preset segmentation rules and the stop word list to obtain the basic text units corresponding to the text to be analyzed and the word items in the basic text units.

6. The method according to claim 1, characterized in that, For each term, determining the term frequency information, the co-occurring word information in the basic discourse unit, and the first quantity information of the basic discourse units in which the term appears includes: Based on the identification information of the basic chapter units, the first dimension is determined; The second dimension is determined based on the terms in the basic text unit; Construct a two-dimensional contingency table based on the first dimension and the second dimension; For each term, the target first dimension and target second dimension are filled into the two-dimensional contingency table to determine the filled two-dimensional contingency table; Based on the filled two-dimensional contingency table, the word frequency information, co-occurring word information in the basic discourse unit, and the first quantity information of the basic discourse unit in which the word appears are determined for each term.

7. The method according to claim 1, characterized in that, The preset high-frequency word conditions include word frequency ranking in the top preset position, and the ratio of the first quantity information to the number of basic chapter units is within a first preset range. The second highest frequency condition includes a word frequency greater than 2 and less than the word frequency of the first preset position, and the first quantity information is within a second preset range.

8. A tag information determining device, characterized in that, include: The first determining module is used to determine the basic discourse unit corresponding to the text to be analyzed and the terms in the basic discourse unit; The second determining module is used to determine, for each term, the term frequency information, the co-occurring word information in the basic discourse unit, and the first quantity information of the basic discourse units in which the term appears; The third determining module is used to determine, among each term, the high-frequency term that meets the preset high-frequency term conditions and the second-highest frequency term that meets the preset second-highest frequency term conditions; The fourth determination module is used to determine the itemset corresponding to the second most frequent term for each high-frequency term, and to use the second quantity information of the basic text units in which the itemset appears as the support count to perform frequent itemset mining and determine the target frequent itemset. The fifth determining module is used to determine the tag information of the text to be analyzed based on the high-frequency terms, the co-occurrence information corresponding to the high-frequency terms, and the target frequent itemset.

9. A computer-readable storage medium storing a computer program for performing the tag information determination method according to any one of claims 1-7.

10. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the tag information determination method according to any one of claims 1-7.

Images