Grouping methods, devices, and media for texts containing similar words

Abstract

This disclosure relates to a method, apparatus, and medium for grouping texts containing similar words. A method for grouping multiple texts containing similar words is provided, comprising: initially grouping the multiple texts, including: calculating the similarity between any two texts in the multiple texts; classifying texts with a similarity greater than a first similarity threshold into an initial group, forming one or more initial groups; if multiple initial groups exist, performing a first-level grouping on the multiple initial groups, including: calculating the similarity between any two initial groups in the multiple initial groups; for each initial group, finding the similar initial group with the highest similarity to that initial group; if the similarity between the initial group and the similar initial group is greater than a second similarity threshold, then classifying the initial group and the similar initial group into a first-level group, forming one or more first-level groups.

Description

Technical Field

[0001] This disclosure generally relates to methods, apparatus, and media for grouping texts containing similar words. Background Technology

[0002] With the rapid development of the information age, the demand for information security is also increasing. Information security protection relies on the classification and grading of assets, making the storage, management, and classification of information particularly crucial. This is especially true in large-scale data industries like telecommunications, where asset inventory management becomes exceptionally challenging. Currently, many knowledge classifications lack sufficient information features, thus involving the classification of similar short phrases or texts containing proper nouns. For example, classifying identical or similar systems across different provinces often requires categorizing by system name. While the system itself may possess other usable information, missing or incorrect information, inaccurate classification, or inconsistent descriptions make direct classification by system name more efficient and accurate.

[0003] Mainstream natural language processing (NLP) methods struggle with classifying short phrases. Most current methods require extensive training samples to develop supervised learning models, or use semantic models to perform vector transformations to calculate pairwise similarities or cluster text. These methods primarily target information-rich articles or short sentences. While processing these longer texts allows for feature extraction—such as TF-IDF (Term Frequency-Inverse Document Frequency) conversion—to create a multi-dimensional semantic space, enabling similarity calculations or feature clustering, these methods are unsuitable for short phrases. Short phrases have fewer segments, and due to their smaller cardinality, they contain more interfering and polysemous words. Term frequency and inverse document frequency features cannot accurately reflect the core meaning of the phrase, especially if it contains proper nouns or has segmentation errors. In such cases, these methods struggle to achieve usable accuracy.

[0004] Classification methods for short phrases primarily rely on supervised learning and require significant upfront resource investment. Currently, there are relatively few methods for classifying / grouping texts or short phrases that lack substantial information but contain proper nouns and industry terminology. The few existing methods rely on business logic or other feature induction (such as part-of-speech tags) or training classification models to rigidly categorize and label short phrases. These methods require large amounts of labeled data, terminology lexicons, business expert knowledge, or parameter tuning by algorithm experts. Furthermore, labeling lacks flexibility (requiring a fixed number of categories); changing the rules necessitates retraining (and sometimes it's unclear how many categories exist). Rigid labeling also struggles to guarantee classification accuracy; checking accuracy requires comparing individual data points, resulting in low efficiency. Summary of the Invention

[0005] According to one aspect of this disclosure, a method for grouping multiple texts containing similar words is provided, comprising: initially grouping the multiple texts, including: calculating the similarity between any two texts in the multiple texts; classifying texts with similarity greater than a first similarity threshold into an initial group, thereby forming one or more initial groups; and, in the presence of multiple initial groups, performing a first-level grouping on the multiple initial groups, including: calculating the similarity between any two initial groups in the multiple initial groups; for each initial group, finding the similar initial group with the highest similarity to the initial group; if the similarity between the initial group and the similar initial group is greater than a second similarity threshold, then classifying the initial group and the similar initial group into a first-level group, thereby forming one or more first-level groups.

[0006] According to another aspect of this disclosure, an apparatus is provided for grouping multiple texts containing similar words, comprising: a memory having instructions stored thereon; and a processor configured to execute the instructions stored in the memory to perform the method described above.

[0007] According to another aspect of this disclosure, a computer-readable storage medium is provided, comprising computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the method described above according to the present disclosure. Attached Figure Description

[0008] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0009] This disclosure will be more clearly understood with reference to the accompanying drawings and the following detailed description, in which:

[0010] Figure 1 A flowchart illustrating a method for grouping multiple texts containing similar words according to an embodiment of the present disclosure is shown.

[0011] Figure 2 An exemplary configuration of a computing device that can implement embodiments of the present disclosure is shown.

[0012] Figure 3 A similarity matrix of short word groups according to one embodiment of the present disclosure is shown.

[0013] Figure 4 The hierarchical structure and final sorting result obtained according to one embodiment of the present disclosure are shown. Detailed Implementation

[0014] The following detailed description is based on the accompanying drawings and provides various exemplary embodiments of the present disclosure to aid in a comprehensive understanding. Various details are included in the following description to aid understanding; however, these details are considered exemplary only and not intended to limit the present disclosure, which is defined by the appended claims and their equivalents. The words and phrases used in the following description are intended only to provide a clear and consistent understanding of the present disclosure. Additionally, descriptions of well-known structures, functions, and configurations may have been omitted for clarity and brevity. Those skilled in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0015] To address the problems existing in the aforementioned text or short word group classification methods, this disclosure proposes an innovative unsupervised multi-level classification, grouping, and ranking algorithm. This algorithm can perform multi-level grouping of text or short word groups containing proper nouns without prior knowledge or classification rule settings. It places more similar word groups in the closest possible positions, thus achieving an innovative classification and ranking algorithm that allows for manually customized grouping or optional intelligent grouping. This solves the problems of accuracy, initial category determination, and subsequent category modification associated with hard labeling, and provides greater flexibility and freedom for grouping. It also facilitates the free selection of the strictness level of classification criteria and the checking of the classification accuracy at each level. In situations where current intelligent algorithms cannot achieve extremely high accuracy when short texts or words lack information, this algorithm provides an optional intelligent algorithm. This innovative algorithm, which can achieve a semi-human, semi-intelligent assistance to humans, can still guarantee high accuracy and efficiency without prior knowledge or classification rule settings, significantly reducing the human intervention required for initial manual labeling, lexicon construction, or model training.

[0016] Figure 1A flowchart of a method 100 for grouping multiple texts containing similar words according to an embodiment of the present disclosure is shown. The method 100 mainly includes: performing initial grouping of the multiple texts (step 101); and, if multiple initial groups exist, performing first-level grouping of the multiple initial groups (step 102).

[0017] In one embodiment, the text may be a passage of text in any language containing approximate words (e.g., proper nouns from any one or more fields). In one embodiment, the text may contain one or more short phrases, one or more short sentences. In one embodiment, the text may be the name of a system or a combination of a system name and a subsystem name.

[0018] In one embodiment, step 101 may include: calculating the similarity between any two texts among the plurality of texts; classifying texts with a similarity greater than a first similarity threshold into an initial group, thereby forming one or more initial groups.

[0019] In one embodiment, step 102 may include: calculating the similarity between any two initial groups among the plurality of initial groups; for each initial group, finding the similar initial group with the highest similarity to the initial group; if the similarity between the initial group and the similar initial group is greater than a second similarity threshold, then classifying the initial group and the similar initial group into a first-level group, thereby forming one or more first-level groups.

[0020] Using the method 100 described above, text can be grouped into two levels.

[0021] In one embodiment, method 100 further includes: when there are multiple lower-level groups (e.g., multiple first-level groups in the case of second-level grouping), performing subsequent layer grouping on the multiple lower-level groups, including: calculating the similarity between any two lower-level groups among the multiple lower-level groups; for each lower-level group, finding the similar lower-level group with the highest similarity to the lower-level group; if the similarity between the lower-level group and the similar lower-level group is greater than the corresponding similarity threshold (e.g., first-level similarity threshold, second-level similarity threshold, etc.), then classifying the lower-level group and the similar lower-level group into a higher-level group, thereby forming one or more higher-level groups; and repeating subsequent layer grouping until it is impossible to continue classifying lower-level groups, or all lower-level groups have been classified into the same group.

[0022] The above method 100 can be used to group text into two or more layers.

[0023] In one embodiment, method 100 may further include: sorting each level of grouping in order from top to bottom according to the number of members in the group, the group number, and the similarity level.

[0024] In one embodiment, the similarity threshold used in higher-level grouping steps is lower than the similarity threshold used in lower-level grouping steps. Thus, text can be grouped into multiple levels according to different levels of similarity.

[0025] In one embodiment, the similarity between two texts can be calculated by converting each text into a vector and using the vectors. Text can be converted into vectors using models such as LSI (Latent Semantic Indexing), trained neural networks such as BERT, Doc2Vec, Word2Vec, Fasttext, and any other suitable method.

[0026] In one embodiment, for each group in each layer, group representative words can be generated for each group member in the group, the group representative words can be converted into vectors, and the vectors can be used to calculate the similarity between two groups in each layer.

[0027] In one embodiment, the median or average of the similarity between members in one group and members in another group in each layer can be used as the similarity between the two groups.

[0028] In one embodiment, before performing the initial grouping, keyword extraction can be performed on each text to obtain multiple word groups corresponding to each of the multiple texts. In this case, the similarity between any two texts among the multiple texts can be calculated based on the multiple word groups (e.g., by converting the multiple word groups corresponding to each text into vectors representing that text).

[0029] In one embodiment, the step of keyword extraction for each text may include: removing stop words or high-frequency words, and adding related words to the obtained keywords or replacing the obtained keywords with related words. This can improve the accuracy of grouping at each level.

[0030] Figure 2 An exemplary configuration of a computing device 1200 capable of implementing embodiments of the present disclosure is shown.

[0031] Computing device 1200 is an example of a hardware device capable of applying the methods described above in this disclosure. Computing device 1200 can be any machine configured to perform processing and / or computation. Computing device 1200 can be, but is not limited to, a workstation, server, desktop computer, laptop computer, tablet computer, personal data assistant (PDA), smartphone, in-vehicle computer, or a combination thereof.

[0032] like Figure 2 As shown, computing device 1200 may include one or more components that can be connected to or communicate with bus 1202 via one or more interfaces. Bus 2102 may include, but is not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus. Computing device 1200 may include, for example, one or more processors 1204, one or more input devices 1206, and one or more output devices 1208. The one or more processors 1204 may be any type of processor and may include, but is not limited to, one or more general-purpose processors or dedicated processors (such as dedicated processing chips). Processor 1202 may, for example, be configured to implement method 100 described above. Input device 1206 may be any type of input device capable of inputting information to the computing device and may include, but is not limited to, a mouse, keyboard, touchscreen, microphone, and / or remote controller. Output device 1208 may be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer.

[0033] The computing device 1200 may also include or be connected to a non-transitory storage device 1214, which may be any non-transitory storage device capable of storing data, and may include, but is not limited to, disk drives, optical storage devices, solid-state storage, floppy disks, flexible disks, hard disks, magnetic tapes or any other magnetic media, compressed disks or any other optical media, cache memory and / or any other storage chip or module, and / or any other medium from which a computer may read data, instructions and / or code. The computing device 1200 may also include random access memory (RAM) 1210 and read-only memory (ROM) 1212. ROM 1212 may store executable programs, utilities, or processes in a non-volatile manner. RAM 1210 provides volatile data storage and stores instructions related to the operation of the computing device 1200. The computing device 1200 may also include a network / bus interface 1216 coupled to a data link 1218. Network / bus interface 1216 can be any kind of device or system capable of enabling communication with external devices and / or networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication devices and / or chipsets (such as Bluetooth). TM Equipment, 802.11 equipment, WiFi equipment, WiMax equipment, cellular communication facilities, etc.

[0034] In one embodiment, this disclosure may provide a computer-readable storage medium including executable instructions that, when executed by an information processing device (e.g., a device such as a computer or smartphone), cause the information processing device to perform the method described above for grouping multiple texts containing similar words.

[0035] The following provides an exemplary summary implementation of the method 100 described above. In this example, multiple short word groups are grouped. Clearly, method 100 can group any text containing similar words (i.e., text containing one or more sentences, or text containing one or more phrases, or short word groups containing multiple scattered words).

[0036] Implementation process (Note that the following description uses short phrases as an example of text. The implementation process is similar for other types of text):

[0037] Step 1: Data Acquisition and Preprocessing

[0038] • Input all short word groups that need to be grouped, extract keywords from them, and perform appropriate preprocessing. Specifically, this includes word segmentation (a terminology dictionary can be set up for word segmentation to avoid missegmentation), removal of stop words / high-frequency words, and addition / replacement of related words (related words can be set up using known business terminology association rules / dictionaries, or association rule learning can be used, or a thesaurus can be used).

[0039] • Transform the segmented short word groups / keywords of each short word group into vectors. Then, transform the short word groups into vectors in the semantic space through a semantic model. The semantic model can be a topic model such as LSI, or it can be transformed through word embeddings of existing mature trained neural networks such as BERT.

[0040] It should be emphasized that the keyword extraction and other preprocessing steps described above are optional, not required. That is, this disclosure can directly perform initial grouping on any number of input texts containing similar words, and the initial grouping step may include the step of directly calculating the similarity between any two texts.

[0041] Step 2: Initial high similarity grouping (i.e., initial grouping)

[0042] • Calculate the similarity between all pairs of short phrases using any reasonable method of pairwise similarity calculation (such as cosine similarity, Euclidean distance, etc.) to form a similarity matrix.

[0043] • By setting a high similarity threshold (when the threshold is 1, the keywords are completely identical), any word pair with a similarity higher than this threshold can be considered to be connected to each other. All connected word pairs are grouped into a small group (when the threshold is 1, all words with completely identical keywords are grouped into a small group / treated as a word pair for subsequent grouping (i.e., merging)), forming multiple initial groups.

[0044] Step 3: Merge groups to form higher-level groups

[0045] • Generate a group representative word / vector for each initial group (when the threshold is 1, since the keywords are completely consistent, the group representative word is the word of any member within the group, which is equivalent to treating all word groups with the same keywords as one word group). Treat the group as a word group whose keyword is the group representative word. In this way, the pairwise similarity matrix between groups can be calculated (the pairwise similarity between groups can also be calculated by the pairwise similarity between members within the group).

[0046] For each initial group, find the group with the highest similarity and determine whether the similarity exceeds the preset minimum similarity threshold required for merging at this layer. If it does, merge the two groups.

[0047] • For the new groups after the merge operation, they can be sorted according to the group size (number of members), group number, and maximum similarity.

[0048] Step 4: Repeatedly create multi-level groups

[0049] • Set a minimum similarity threshold for each layer (use a preset threshold if no threshold is set). If there are still new groups with pairwise similarity exceeding the threshold, repeat step three to merge the new groups again until the loop termination condition is met, forming multi-layer groups.

[0050] • After merging and grouping at each level, the word groups are sorted and output according to the group size (number of members), group number, and similarity level, from top to bottom. Users can choose whether to enable automatic classification and tagging.

[0051] The following is merely an example of a more specific implementation method for steps two, three, and four above.

[0052] Step A: Initial high similarity grouping

[0053] 1. Generate the original short word phrase similarity matrix.

[0054] Suppose there are N short phrases, then we have: Figure 3 The similarity matrix shown.

[0055] a) Similarity calculation can use any mature NLP model.

[0056] It includes two steps:

[0057] 1) After grouping short words into words, map them to a vector space (this can be done using a topic model such as LSI, or by converting them using word embeddings from existing mature trained neural networks such as BERT).

[0058] 2) Calculate the similarity of short word phrase vectors (the calculation method can be cosine similarity, Euclidean distance, etc.).

[0059] b) Replace the similarity between the phrase itself and itself with -1, i.e., sim(i,i)=-1,i∈(0,N)

[0060] 2. Extract all short word pairs with a similarity higher than the set minimum similarity threshold T, and sort them from high to low similarity.

[0061] 3. Initial high similarity grouping: Short word groups with pairwise similarity higher than T are grouped together.

[0062] Suppose there are N short word groups, and T is the minimum similarity threshold (only completely similar words are considered a group when the threshold is 1). High initial similarity requires high similarity, so it is recommended that the threshold T be set to a value greater than 0.8 and less than or equal to 1.

[0063] The initial grouping formula is:

[0064] y i = -1, i∈(1,N)

[0065] for If sim(i,j)≥T

[0066] So

[0067] After grouping once by sorting, repeat the grouping process until the groups remain unchanged.

[0068] Step 2: Merge groups to form new parent groups

[0069] 1. Calculation of inter-group similarity

[0070] Similarity between groups can be calculated in (but is not limited to) the following two ways:

[0071] a) The median or average of pairwise similarity among individual group members

[0072] Suppose we have group A and group B:

[0073]

[0074] SimAB = Median(Sim(i,j))(median value)

[0075] (average value)

[0076] Where |A| represents the number of members in group A, and |B| is similar.

[0077] b) Generate group representative segments from group member segments, and use the group representative segments to calculate pairwise similarity (the calculation method is consistent with the previous similarity calculation between word groups).

[0078] Suppose there are N word groups in a group. The set of word segments that have been processed (word segmentation and stop word removal) can be called {word segment i}, where i∈(1,N).

[0079] Group A represents the word segment {segment A}, which can be represented by (but not limited to) the following three methods (in descending order of strictness):

[0080] a) Intersection: {Word segment A} = {Word segment 1} ∩ {Word segment 2} ∩ ... ∩ {Word segment N}

[0081] b) Frequent word set: {All word segments that appear in more than x% (e.g., 50%) of this word group}

[0082] c) Union: {segment A} = {segment 1} ∪ {segment 2} ∪ ... ∪ {segment N}

[0083] 2. Subsequent higher-level groups are merged and grouped.

[0084] Assuming there are M subgroups, group G in the similarity matrix... i With group G j The similarity between them is represented as Sim(G) i G j ), group G of layer L i The grouping is represented as

[0085] For each lower-level group G i Find all lower-level groups with the highest similarity (and the highest similarity must be higher than the minimum similarity threshold T set for the parent level). L The algorithm then sorts all extracted lower-level groups with the highest similarity from highest to lowest similarity (the minimum similarity threshold T for merging each level). L The settings can be adjusted through experience, manual debugging based on the final classification results, or manual or automated parameter tuning through some metrics. Feasible metrics include (but are not limited to): the sum of the average similarity between all points within a group and the average similarity between all points within a group and all points outside the group. The larger this metric is, the more reasonable the grouping is, that is, the average similarity within a group is much larger than the average similarity between points within a group and points outside the group.

[0086] The grouping formula for the Lth layer can then be expressed as:

[0087]

[0088] for satisfy:

[0089]

[0090]

[0091] After grouping once by sorting, repeat the grouping process until the groups remain unchanged.

[0092] 3. After merging and grouping at each level, sort and output the word groups in order from the upper level to the lower level according to the group size (number of group members), group number, and similarity.

[0093] 4. Repeat steps 1, 2, and 3 until the termination condition is met.

[0094] Termination conditions can be set in two ways (but are not limited to):

[0095] a) When there are no pairs of similarities greater than the minimum merge similarity threshold T for that level. L The lower-level grouping pair of (sim(G)) i G maxi )≥T L When this condition is met, the loop automatically terminates, indicating that merging can no longer continue (this is the highest level).

[0096] b) All phrases have been merged into the same group, and it is determined that no further merging is possible (this is the highest level).

[0097] This disclosure presents a multi-level classification and grouping method for texts or short phrases containing proper nouns. It addresses the problems of existing technologies requiring excessive prior human intervention in training and debugging (designing models or rules requires prior knowledge such as terminology lexicons and business knowledge, or a large number of labeled samples for supervised learning training), and the issues arising from the need for pre-defined classifications in supervised classification models, leading to problems with accuracy, initial category determination, and subsequent category modification. This method achieves efficient multi-level grouping of approximate short phrases containing proper nouns, providing greater flexibility and freedom in grouping. It allows for easier selection of the strictness of classification criteria and checking the accuracy of each level. It offers an alternative intelligent algorithm when current intelligent algorithms cannot achieve high accuracy with short words lacking information. This innovative algorithm, which can achieve a semi-human, semi-intelligent approach, maintains high accuracy and efficiency without requiring prior knowledge or classification rule settings, significantly reducing the human intervention needed for initial manual labeling, lexicon construction, or model training.

[0098] The following is merely an example describing the specific classification situation when the method of this disclosure is applied to the classification and grading of the system (the following data is extracted from real data after anonymization and simple modification, **mostly regional, and stop words will be excluded).

[0099] 1. Data preprocessing and initial grouping:

[0100]

[0101] Table 1

[0102] In Table 1 above, in the rightmost column, the initial group number for the top two rows is 1, the initial group number for the third and fourth rows is 2, the initial group number for the fifth and sixth rows is 3, and so on, numbered 4, 5, 6, 7, 8, 9, 10. The total number of groups after the initial grouping is 10.

[0103] * The removed stop words include all high-frequency data words (such as 'center', 'platform', 'system') and words describing regions (such as provinces and cities, 'company').

[0104] * You can customize the related keywords dictionary and add rules and conditions: for example, if 'CRM' or 'customer relations' appears, replace both words with one of them, or include both words. Here, selecting to include both increases keyword weight.

[0105] 2. Generation of first-level grouping and group words in the parent category.

[0106]

[0107] Table 2

[0108] 3. Generation of second-level grouping and group terminology.

[0109]

[0110] Table 3

[0111] * Semantic similarity considers not only whether the word segments are the same, but also the importance of the word segments and the relationship between the words.

[0112] * The difference between the parent group and the initial group is that in the parent group, each group only selects the group with the highest similarity for merging (assuming that there are multiple groups with similarity exceeding the threshold).

[0113] 4. Third-level grouping at the higher level

[0114]

[0115] Table 4

[0116] *The termination condition is met here: all phrases have been merged into the same group, and no further merging is possible.

[0117] 5. Hierarchical structure and final sorting

[0118] Figure 4 The hierarchical structure and final order are shown, where each small circle represents a short phrase.

[0119] Final initial grouping order (see) Figure 4The rightmost group is: Group 2, Group 5, Group 6, Group 8, Group 9, Group 10, Group 1, Group 3, Group 4, and Group 7 (Group 1, Group 2, and Group 3 each have two short word groups. Since this example uses preset / identical keywords, the initial group order can be arbitrary. When the keywords are different, the similarity relationship with the last word of the previous group can be used as the ordering criterion). The final ordering makes the more similar short word groups cluster together, which is convenient for manual inspection and to determine which layer of classification is used as the standard for automatic classification and labeling.

[0120] The following six technical points are important components of this disclosure:

[0121] • The connection relationships between texts are determined by setting a threshold and calculating pairwise similarity, and the text connection relationships are used for initial grouping.

[0122] • Use innovative concepts such as group-based word segmentation or text (or calculate pairwise similarity between groups based on pairwise similarity of each group member) to perform pairwise similarity calculations between groups.

[0123] • By using pairwise similarity between groups and setting thresholds for each level, lower-level groups are merged to form higher-level groups, thus creating multi-level grouping.

[0124] • By using multi-level grouping and sorting by group number and similarity level from top to bottom, the goal is to place the most similar words in the most recent position.

[0125] • Concepts of multi-level grouping / classification rules and custom grouping: Users can choose to use grouping rules at any level for automatic / intelligent tagging and classification, or manually classify by the final sorting to achieve a human-assisted function (since the final sorting places the most similar word groups in the closest position, it is more convenient for manual grouping).

[0126] • Improve grouping accuracy through preprocessing of text or short phrases: remove stop words (including high-frequency words) and add or replace related words.

Claims

1. A method for grouping multiple texts containing similar words, comprising: Initial grouping of the multiple texts includes: Calculate the similarity between any two texts from the plurality of texts; Texts with a similarity greater than the first similarity threshold are grouped into an initial group, thereby forming one or more initial groups; In the presence of multiple initial groups, the multiple initial groups are grouped into a first-level group, including: Calculate the similarity between any two initial groups from the plurality of initial groups; For each initial group, find the similar initial group with the highest similarity to the initial group. If the similarity between the initial group and the similar initial group is greater than the second similarity threshold, then the initial group and the similar initial group are classified into a first-level group, thereby forming one or more first-level groups. In the presence of multiple lower-level groups, these lower-level groups are further grouped into subsequent layers, including: Calculate the similarity between any two lower-level groups among the plurality of lower-level groups; For each lower-level group, find the most similar lower-level group. If the similarity between the lower-level group and the similar lower-level group is greater than the corresponding similarity threshold, then classify the lower-level group and the similar lower-level group into a higher-level group, thereby forming one or more higher-level groups; and Repeat the grouping process for subsequent layers until it is no longer possible to classify lower-level groups, or all lower-level groups have been classified into the same group, wherein the similarity threshold used in the higher-level grouping steps is lower than the similarity threshold used in the lower-level grouping steps.

2. The method according to claim 1, further comprising: For each level of grouping, sort them in order from the upper level to the lower level according to the number of members in the group, the group number, and the degree of similarity.

3. The method according to claim 1, wherein, The similarity between two texts is calculated by converting each text into a vector and then using the vectors.

4. The method according to claim 1, wherein, For each group in each layer, a group representative word is generated for that group, the group representative word is converted into a vector, and the vector is used to calculate the similarity between two groups in each layer. The group representative word for each group is the intersection of the word segments of each member of that group.

5. The method according to claim 1, wherein, The median or average similarity between members in one group and members in another group in each layer is taken as the similarity between the two groups.

6. The method according to claim 1, further comprising: Before the initial grouping, keyword extraction is performed on each text to obtain multiple word groups corresponding to the multiple texts, and The similarity between any two texts in the plurality of texts is calculated based on the plurality of word groups.

7. The method according to claim 6, wherein keyword extraction processing for each text includes: Remove stop words or high-frequency words, and add related words to the obtained keywords or replace the obtained keywords with related words.

8. An apparatus for grouping multiple texts containing similar words, comprising: A memory that stores instructions; as well as The processor is configured to execute instructions stored in the memory to perform the method according to any one of claims 1 to 7.

9. A computer-readable storage medium comprising computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 7.

Images