Text classification methods, text search methods, devices, equipment and media
By calculating similar scores and common words between text titles and automatically classifying categories, the problem of low efficiency in text data classification is solved, and efficient text classification and annotation is achieved.
Patent Information
- Application Number
- CN202411045386.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-31
AI Technical Summary
The prior art is difficult to efficiently classify and manage large amounts of text data, especially in the fields of medical and official documents, resulting in inefficient and high cost.
By obtaining text titles for word segmentation processing, compute similar scores between titles, find candidate titles with the largest similar score, and classify categories based on common words, realize automatic text classification and reduce manual participation.
It improves the efficiency of text classification, reduces labor and time costs, and realizes efficient classification and annotation of multiple texts.
Smart Images

Figure CN118568264B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of text classification, and in particular to a text classification method, a text search method, a text classification device, a text search device, an electronic device, and a computer-readable storage medium. Background Art
[0002] In various fields, a large number of texts are usually involved. For example, medical texts may include texts of different medical subjects, and official documents may include texts of different departments. Due to the large number and sources of these texts, how to classify the texts is also an urgent problem to be solved. Summary of the invention
[0003] The embodiments of the present disclosure at least provide a text classification method, a text search method, a text classification device, a text search device, an electronic device and a computer-readable storage medium, which can realize the classification of texts.
[0004] The present disclosure provides a text classification method, including:
[0005] Obtaining titles of multiple texts, performing word segmentation processing on each of the titles to obtain word segmentation results, and determining similarity scores between any two titles based on the word segmentation results of each of the titles;
[0006] Determine the two first target titles corresponding to the maximum similarity scores as titles under the same category, search for a candidate title with the maximum sum of similarity scores with the two first target titles from other titles except the two first target titles, and classify the candidate title into the category corresponding to the first target title if the common words between the candidate title and the titles that have been classified meet the first preset requirement, return to the step of searching for the candidate title with the maximum sum of similarity scores with the two first target titles, and stop searching until the common words between the candidate title and the titles that have been classified do not meet the first preset requirement;
[0007] For the remaining titles except the titles that have been divided into categories, return to the step of determining the two first target titles corresponding to the maximum similarity scores as titles under the same category until the remaining titles are respectively divided into corresponding categories;
[0008] Based on the common words contained in each title under each category, a category label of each category is determined, and based on the category label, the text corresponding to the title under the category is categorized.
[0009] In the disclosed embodiment, the two first target titles corresponding to the maximum similarity score are determined as titles under the same category, and then the candidate title with the largest sum of similarity scores with the two first target titles is searched. It can be understood that if the similarity score is the largest and the common words between the candidate title and the titles that have been divided into categories meet the first preset requirement, it means that the candidate title is relatively similar to the two first target titles. Therefore, the candidate title can be classified into the category corresponding to the first target title, and then the next candidate title with the largest sum of similarity scores can be searched. If the common words between the candidate title currently being searched and the titles that have been divided into categories do not meet the first preset requirement, it means that the candidate title does not belong to the category, and the title search for the category can be stopped. Then other categories can be determined in turn in a loop until the remaining titles are divided into corresponding categories. In this way, the category division can be completed for each title. Finally, for each category, the text corresponding to the title under the category is labeled based on the category label of the category, thereby realizing the classification of multiple texts.
[0010] Furthermore, in the above text classification process, no human intervention is required, which is conducive to reducing labor costs and time costs, thereby improving the efficiency of text classification.
[0011] In an optional implementation manner, determining the similarity score between any two titles according to the word segmentation result of each of the titles includes:
[0012] According to the plurality of word segmentation results, the plurality of titles are divided into a plurality of title sets, and according to the number of titles included in each of the title sets, a set priority arrangement order between the title sets is determined;
[0013] A target title set is determined from the plurality of title sets in sequence based on the set priority ranking order, and a similarity score between any two titles in the target title set is determined.
[0014] In the disclosed embodiment, by determining the set priority arrangement order between the various title sets, the target title set can be determined in turn from the multiple title sets according to the set priority arrangement order, and the similarity score between any two titles in the target title set can be determined. It can be understood that since the number of titles in the title set with a high priority is also large, if the same title is included in other title sets with lower priorities, there is no need to repeatedly calculate the similarity score, which is conducive to saving computing resources.
[0015] In an optional implementation manner, dividing the multiple titles into multiple title sets according to the multiple word segmentation results includes:
[0016] Determining multiple groups of target segmented words that meet the second preset requirement from the multiple segmented word results, and generating a title set for each group of target segmented words based on target titles containing the target segmented words among the multiple titles, wherein the multiple groups of target segmented words are at least partially different from each other;
[0017] Based on the title sets corresponding to the multiple groups of target word segments, the multiple title sets are obtained.
[0018] In the disclosed embodiment, multiple groups of target segmented words that meet the second preset requirement are determined, and for each group of target segmented words, multiple titles are grouped to generate a title set, which facilitates subsequent determination of similarity scores for the titles in the title set.
[0019] In an optional implementation manner, performing word segmentation on each of the titles to obtain a word segmentation result includes:
[0020] Perform word segmentation processing on each of the titles to obtain an initial word segmentation result;
[0021] The word segmentation results are obtained by removing the word segmentations that do not meet the third preset requirement in the initial word segmentation results.
[0022] In the embodiment of the present disclosure, by removing the segmented words that do not meet the third preset requirement in the initial segmented word results, the words in the segmented word results can better represent the subject of the title.
[0023] In an optional implementation manner, determining the category label of each category based on the common words contained in each title under each category includes:
[0024] Determining a target category from the plurality of categories, and determining a category label of the target category based on common words included in the titles corresponding to the target category;
[0025] Based on the category label of the target category, marking titles under other categories among the multiple categories except the target category;
[0026] For each of the other categories, a category label of the other category is determined based on common words contained in unlabeled titles under the other category.
[0027] In the disclosed embodiment, since titles under different categories may also have common words, after determining the category label of the target category, the titles under other categories are marked based on the category label of the target category. These marked titles will not participate in the subsequent process of determining the category label. In this way, the number of titles used to determine the category labels of other categories can be reduced, thereby improving the efficiency of determining the category labels of other categories.
[0028] In an optional implementation, determining the target category from the multiple categories includes:
[0029] According to the number of titles corresponding to each of the categories, the multiple categories are prioritized to obtain a category priority arrangement order among the multiple categories, and according to the category priority arrangement order, the target category is determined from the multiple categories.
[0030] In the disclosed embodiment, multiple categories are prioritized according to the number of titles corresponding to each category. In this way, when at least one target category is determined according to the category priority order, the category containing more titles can be preferentially selected as the target category. In this way, in the subsequent process of determining the category labels of each category, other categories can be marked as much as possible, thereby reducing the number of titles.
[0031] In an optional implementation manner, for each of the other categories, determining the category label of the other category based on the common words contained in the unlabeled titles under the other category includes:
[0032] For each of the other categories, when the number of unlabeled titles under the other categories is not less than a first preset number, the category label of the other category is determined based on the common words contained in the unlabeled titles under the other categories.
[0033] In the disclosed embodiment, if the number of unmarked titles under other categories is not less than the first preset number, it means that the unmarked titles have other common words that can represent the subject of the title. Therefore, the category label of the unmarked titles can be determined based on the common words contained in the unmarked titles. Similarly, if the number of unmarked titles under other categories is less than the first preset number, it means that the unmarked titles have little impact on the overall classification process, and there is no need to determine their category labels, which can save resources.
[0034] The present disclosure also provides a text search method, including:
[0035] Obtaining a text whitelist; the text whitelist includes at least one preset whitelist word;
[0036] Based on the at least one whitelist word, a target text containing the at least one whitelist word is searched from the text stored in the text library; the text library corresponds to multiple category labels, each of the category labels corresponds to multiple texts, and the multiple texts corresponding to each of the category labels are determined based on the text classification method described in any of the aforementioned embodiments.
[0037] In the disclosed embodiment, by obtaining a text whitelist and searching the text in the text library based on the whitelist words in the text whitelist, the search range can be reduced. In addition, the multiple texts corresponding to each category label in the text library are determined based on the text classification method, so that the text in the text library is more standardized, so that when searching, the text under each category label can be searched in a targeted manner according to the text whitelist, which is conducive to improving the search efficiency.
[0038] In an optional implementation manner, searching for a target text containing the at least one whitelist word from texts stored in a text library based on the at least one whitelist word comprises:
[0039] Acquire a text blacklist; the text blacklist includes at least one preset blacklist category;
[0040] Based on the at least one blacklist category, shielding and marking texts corresponding to the same category label as the at least one blacklist category in the text library;
[0041] Based on the at least one whitelist word, the target text is searched from texts in the text library that are not marked with a blocking mark.
[0042] In an embodiment of the present disclosure, texts in a text library corresponding to category labels that are the same as at least one blacklist category in a text blacklist are shielded and marked, so that the shielded and marked texts will not be processed, thereby reducing the number of texts in the text library. Furthermore, target texts are searched from texts that have not been shielded and marked based on at least one whitelist word, thereby improving search efficiency.
[0043] In an optional implementation manner, the text whitelist is generated by the following method:
[0044] For each category label, determining a target word from the text content of each text corresponding to the category label; the target word refers to the same word whose number of words in the text is greater than a second preset number;
[0045] Determine a target word set corresponding to the category label based on the target words corresponding to each text corresponding to the category label;
[0046] The target word sets corresponding to each category label are compared respectively, and the target words in each target word set that do not meet the fourth preset requirement are removed to obtain a new target word set, and based on the target words in the new target word set whose number is greater than the third preset number, the text whitelist is generated.
[0047] Here, the target words may refer to repeated words determined from each text. Thus, for each category label, the target word set (i.e., repeated word set) corresponding to the category label may be determined based on the target words determined in the corresponding texts. By removing target words that do not meet the fourth preset requirements, the conciseness and accuracy of the new target word set may be improved.
[0048] In an optional implementation manner, determining the target word from the text content of each text corresponding to the category label includes:
[0049] For each text corresponding to the category label, construct a position index for each character in the text, and based on the position index of each character, determine multiple position indexes corresponding to the same target character;
[0050] For each target character, any two position indexes of the target character are combined to obtain an index array;
[0051] When the index arrays respectively corresponding to at least two target characters meet the fifth preset requirement, the word composed of the at least two target characters is determined as the target word.
[0052] In the disclosed embodiment, the target word can be determined by the above-mentioned non-segmentation method, thus improving the accuracy of determining the target word.
[0053] The present disclosure also provides a text classification device, including:
[0054] A text title acquisition module is used to acquire titles of multiple texts, perform word segmentation processing on each of the titles, obtain word segmentation results, and determine the similarity score between any two titles based on the word segmentation results of each of the titles;
[0055] A first title classification module is used to determine the two first target titles corresponding to the maximum similarity scores as titles under the same category, and to search for a candidate title with the maximum sum of similarity scores with the two first target titles from other titles other than the two first target titles, and to classify the candidate title into the category corresponding to the first target title if the common words between the candidate title and the titles that have been classified meet the first preset requirement, and to return to the step of searching for the candidate title with the maximum sum of similarity scores with the two first target titles until the common words between the candidate title and the titles that have been classified do not meet the first preset requirement, and the search is stopped;
[0056] A second title division module is used for returning to the step of determining the two first target titles corresponding to the maximum similarity scores as titles under the same category for the remaining titles among the multiple titles except the titles that have been divided into categories, until the remaining titles are respectively divided into corresponding categories;
[0057] The category label determination module is used to determine the category label of each category based on the common words contained in each title under the category, and to perform category marking on the text corresponding to the title under the category based on the category label.
[0058] The present disclosure also provides a text search device, including:
[0059] An acquisition module, used to acquire a text whitelist; the text whitelist includes at least one preset whitelist word;
[0060] A search module is used to search for a target text containing the at least one whitelist word from texts stored in a text library based on the at least one whitelist word; the text library corresponds to a plurality of category labels, each of the category labels corresponds to a plurality of texts, and the plurality of texts corresponding to each of the category labels are determined based on any of the above-mentioned text classification methods.
[0061] An embodiment of the present disclosure also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of any possible text classification method mentioned above or the steps of any possible text search method mentioned above are performed.
[0062] The embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program executes the steps of any possible text classification method or any possible text search method.
[0063] For a description of the effects of the above-mentioned text classification device, text search device, electronic device and computer-readable storage medium, please refer to the description of the above-mentioned text classification method and text search method, which will not be repeated here.
[0064] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure.
[0065] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0067] Figure 1 A flowchart of a text classification method provided by an embodiment of the present disclosure is shown;
[0068] Figure 2 A flow chart of a method for determining a category label for each category provided by an embodiment of the present disclosure is shown;
[0069] Figure 3 A flowchart of another text classification method provided by an embodiment of the present disclosure is shown;
[0070] Figure 4 A flowchart of a text search method provided by an embodiment of the present disclosure is shown;
[0071] Figure 5 A schematic diagram of the structure of a text classification device provided by an embodiment of the present disclosure is shown;
[0072] Figure 6 A schematic diagram of the structure of a text search device provided by an embodiment of the present disclosure is shown;
[0073] Figure 7 A schematic diagram showing the structure of another text search device provided by an embodiment of the present disclosure is shown;
[0074] Figure 8 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0075] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0076] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0077] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.
[0078] Research has found that a large number of texts are usually involved in various fields. For example, medical texts may include texts from different medical subjects, and official documents may include texts from different departments. Due to the large number and sources of these texts, how to classify the texts is also an urgent problem to be solved.
[0079] In addition, in the related technology, manual classification is also used to classify texts. The above classification method has the problem of high manpower cost and time cost, which leads to low classification efficiency.
[0080] Based on the above research, the present disclosure provides a text classification method, firstly obtaining titles of multiple texts, and performing word segmentation processing on each of the titles to obtain word segmentation results, and determining the similarity score between any two titles according to the word segmentation results of each of the titles; secondly determining the two first target titles corresponding to the maximum similarity score as titles under the same category, and searching for candidate titles with the maximum sum of similarity scores between the two first target titles from titles other than the two first target titles, and if the common words between the candidate titles and the titles that have been divided into categories meet the first preset requirement, classifying the candidate titles into the category corresponding to the first target titles, and returning the result to be executed. The step of searching for a candidate title with the largest sum of similarity scores with the two first target titles is performed until the common words between the candidate title and the titles that have been classified do not meet the first preset requirement, and the searching is stopped; then, for the remaining titles except the titles that have been classified among the multiple titles, returning to the step of determining the two first target titles corresponding to the largest similarity scores as titles under the same category, until the remaining titles are respectively classified into corresponding categories; finally, for each of the categories, based on the common words contained in the titles under the category, the category label of the category is determined, and the text corresponding to the titles under the category is labeled based on the category label.
[0081] In the disclosed embodiment, the two first target titles corresponding to the maximum similarity score are determined as titles under the same category, and then the candidate title with the largest sum of similarity scores with the two first target titles is searched. It can be understood that if the similarity score is the largest and the common words between the candidate title and the titles that have been divided into categories meet the first preset requirement, it means that the candidate title is relatively similar to the two first target titles. Therefore, the candidate title can be classified into the category corresponding to the first target title, and then the next candidate title with the largest sum of similarity scores can be searched. If the common words between the candidate title currently being searched and the titles that have been divided into categories do not meet the first preset requirement, it means that the candidate title does not belong to the category, and the title search for the category can be stopped. Then other categories can be determined in turn in a loop until the remaining titles are divided into corresponding categories. In this way, the category division of each title can be completed. Finally, for each category, the text corresponding to the title under the category is labeled based on the category label of the category, thereby realizing the classification of multiple texts.
[0082] Furthermore, in the above text classification process, no human intervention is required, which is conducive to reducing labor costs and time costs, thereby improving the efficiency of text classification.
[0083] To facilitate understanding of this embodiment, a text classification method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the text classification method provided in the embodiment of the present disclosure is generally an electronic device, and the electronic device can be a server, which can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data and artificial intelligence platforms. In other implementations, the electronic device can also be a terminal device, wherein the terminal device can be a mobile device, a user terminal, a terminal, a handheld device, a computing device, a vehicle-mounted device, a wearable device, and the like.
[0084] In other implementations, the method can also be applied to an implementation environment consisting of a terminal device and a server. In addition, the text classification method can also be implemented by a processor calling a computer-readable instruction stored in a memory.
[0085] A text classification method provided by an embodiment of the present disclosure is described below with reference to the accompanying drawings.
[0086] See also Figure 1 As shown, it is a flowchart of a text classification method provided by an embodiment of the present disclosure, such as Figure 1 As shown in , the method includes steps S101 to S104, wherein:
[0087] S101: Acquire titles of multiple texts, perform word segmentation processing on each of the titles to obtain word segmentation results, and determine similarity scores between any two titles based on the word segmentation results of each of the titles.
[0088] Here, the text may be a text in any field, for example, it may be a text in the medical field, the educational field, the literary field, etc., and is not limited here. For example, in the medical field, the text may be a text of different medical subjects; in the educational field, the text may be a text published by different educational departments, etc.; in the literary field, the text may be a novel, a poem, etc.
[0089] It can be understood that since the title is a short sentence that represents the content of the text (or article, work, etc.), the text classification in the subsequent steps can be performed based on the text title.
[0090] Specifically, after obtaining the titles of multiple texts, each title can be segmented to obtain a segmentation result.
[0091] For example, if Title 1 is "Environmental governance is the current work focus", it will be segmented and the result will be "environment, governance, for, current, work, focus"; if Title 2 is "Work guiding opinions on environmental governance", it will be segmented and the result will be "environment, governance, of, work, guidance, opinions".
[0092] In some embodiments, when each of the titles is segmented to obtain a segmentation result, the following steps may be included: each of the titles is segmented to obtain an initial segmentation result, and then the segmentations in the initial segmentation result that do not meet the third preset requirements are removed to obtain the segmentation result.
[0093] Among them, the segmentations that do not meet the third preset requirement can be understood as common words that have no impact on the category classification. The segmentations that do not meet the third preset requirement may refer to meaningless common words whose frequency of occurrence is greater than the first frequency. Here, the segmentations that do not meet the third preset requirement may include quantifiers, conjunctions, public data sets, etc. For example, quantifiers may include "one", conjunctions may include "of", "and", etc., and public data sets may include public data of some websites (such as public survey data, experimental data, etc.).
[0094] In this way, after obtaining the word segmentation result of each title, the similarity score between any two titles can be determined according to the word segmentation result of each title.
[0095] In some implementations, when determining the similarity score between any two titles based on the word segmentation results of each of the titles, the similarity score can be determined by the following steps (1) to (2):
[0096] (1) For any two titles, based on the word segmentation results corresponding to the two titles, determine the common words between the two titles, and merge the common words to obtain two merged phrases; the two merged phrases are respectively included in the two titles, and each merged phrase includes at least one merged word.
[0097] As can be seen from the foregoing, the word segmentation process has been completed in step S101, and the word segmentation results of each title have been obtained. Therefore, the common words between the two titles can be determined based on the word segmentation results. After the common words are determined, the common words can be merged to obtain two merged phrases.
[0098] For example, Title A is "Complete work within the scope of responsibilities", and the word segmentation result of Title A is: Complete work within the scope of responsibilities; Title B is "Perform work within the scope of responsibilities", and the word segmentation result of Title B is: Perform work within the scope of responsibilities, then it can be determined that the common word between Title A and Title B is "Work within the scope of responsibilities", and then the above common words are merged to obtain merged phrase C: scope of responsibilities, work, and merged phrase D: work within the scope of responsibilities, that is, merged phrase C includes the merged words "scope of responsibilities" and "work", and merged phrase D includes the merged word "work within the scope of responsibilities".
[0099] (2) Determining the similarity score based on the length of each of the merged words in the two merged word groups, the weight of the length of each merged word, and the lengths of the two titles.
[0100] The length of the merged word may refer to the number of characters in the merged word. For example, for the merged word “scope of responsibilities”, its length is 4, and for the merged word “work”, its length is 2.
[0101] In this implementation, different lengths correspond to corresponding weights. Specifically, the weight corresponding to a merged word with a length of 1 is 0.3, the weight corresponding to a merged word with a length of 2 or 3 is 1, the weight corresponding to a merged word with a length of 4 is 1.2, the weight corresponding to a merged word with a length of 5 is 1.5, and the weight corresponding to a merged word with a length greater than 5 is 2.
[0102] The length of the title refers to the number of characters in the title. For example, based on the above example, Title A is "Complete work within the scope of responsibilities" and its title length is 9, and Title B is "Perform work within the scope of responsibilities" and its title length is 8.
[0103] Please refer to formula (1), which is an expression for determining the similarity score in the embodiment of the present disclosure:
[0104] (1)
[0105] in, is the i-th combined word, For The corresponding weights, is the title length of the first title, The title length of the second title.
[0106] Continuing with the above example, we can determine the similarity score between title A "Complete work within the scope of responsibilities" and title B "Perform work within the scope of responsibilities" is .
[0107] It should be noted that the larger the similarity score, the more similar the two titles are.
[0108] S102: Determine the two first target titles corresponding to the maximum similarity score as titles under the same category, and search for a candidate title with the largest sum of similarity scores with the two first target titles from other titles except the two first target titles, and classify the candidate title into the category corresponding to the first target title if the common words between the candidate title and the titles that have been classified meet the first preset requirement, and return to the step of searching for the candidate title with the largest sum of similarity scores with the two first target titles until the common words between the candidate title and the titles that have been classified do not meet the first preset requirement, and stop searching.
[0109] The candidate title may refer to a candidate title that belongs to the same category as the first target title.
[0110] The common words between the candidate title and the categorized titles respectively meeting the first preset requirement may mean that the candidate title and the categorized titles have the same common words (unlimited in number), and the same common words are words with practical meanings for each title.
[0111] For example, the first target title A is: Key information must be paid special attention to, and the first target title B is: Key information must be paid attention to, and the common words of these two first target titles include: key, information, must, and attention;
[0112] Candidate title C is: The information is very important and must be paid attention to. The common words between candidate title C and the first target titles A and B are: information, must, and attention.
[0113] Candidate title D is: Must be very happy, and the common words between candidate title D and the first target titles A and B and candidate title C are: Must;
[0114] Since the common word “must be” has no practical meaning for the title, the candidate title D “must be very happy” will not be classified into this category.
[0115] As can be seen from the previous text, the larger the similarity score, the more similar the two titles are. Therefore, it can be considered that the two titles correspond to the same category. Therefore, the two titles corresponding to the maximum similarity score (that is, the first target titles) can be determined as titles under the same category, and then from other titles except the two first target titles, find the candidate title with the largest sum of similarity scores with the two first target titles.
[0116] Here, since a title usually includes multiple words, even if there is a title among other titles that has the largest sum of similarity scores with the two first target titles, it cannot be directly classified. The reason may be that some meaningless words in the title make the similarity score the largest, such as some degree words, "certain", "approximately", etc., or it may be due to different common words that make the similarity score the largest. For example, the common word between title C and the first target title A is D, and the common word between title C and the first target title B is E. Although the common words are different, title C will also be the title with the largest sum of similarity scores with the two first target titles.
[0117] Therefore, in this embodiment, after finding a candidate title, it is also necessary to determine whether the common words between the candidate title and the titles that have been classified meet the first preset requirements. If they meet the first preset requirements, the candidate title is classified into the category corresponding to the first target title.
[0118] Then continue to search for the next candidate title, that is, return to the step of searching for the candidate title with the largest sum of similarity scores between the two first target titles. If the common words between the candidate title currently being searched and the titles that have been divided into categories do not meet the first preset requirement, it means that the candidate title does not belong to the category, then stop searching for titles in this category and search for the next category.
[0119] S103: For the remaining titles among the multiple titles except the titles that have been classified, return to the step of determining the two first target titles corresponding to the maximum similarity scores as titles under the same category until the remaining titles are respectively classified into corresponding categories.
[0120] It can be understood that in step S102, other titles belonging to the same category as the first target title can be divided, but there will still be remaining titles that have not been divided. Therefore, for the remaining titles, the step of "determining the two first target titles corresponding to the maximum similarity score as titles under the same category" in step S102 is returned to execution, that is, the next category is continued to be divided here until all remaining titles are divided into corresponding categories.
[0121] For example, if the remaining titles include titles C, D, E, F, and G, and the similarity score between C and D is the largest, then C and D are classified into the same category. If the sum of the similarity scores between title E and C and D is the largest, then title E is used as a candidate title. If the common words between title E and C and D meet the first preset requirement, then E is classified into the category to which C and D belong. Similarly, for titles F and G, the category classification is also performed based on the above method.
[0122] S104: For each of the categories, determine a category label of the category based on common words contained in each title under the category, and perform category labeling on text corresponding to the title under the category based on the category label.
[0123] It can be understood that the common words contained in the various titles belonging to the same category are usually common words that can express the core of the title. Therefore, for each category, the category label of the category can be determined based on the common words contained in the various titles under the category. In this way, after determining the category label, the text corresponding to the title under the category can be categorized based on the category label, thereby realizing the classification of multiple texts.
[0124] In the embodiment of the present disclosure, the two first target titles corresponding to the maximum similarity score are determined as titles under the same category, and then the candidate title with the largest sum of similarity scores with the two first target titles is searched. It can be understood that if the similarity score is the largest and the common words between the candidate title and the titles that have been divided into categories meet the first preset requirement, it means that the candidate title is relatively similar to the two first target titles. Therefore, the candidate title can be classified into the category corresponding to the first target title, and then the next candidate title with the largest sum of similarity scores can be searched. If the common words between the candidate title currently being searched and the titles that have been divided into categories do not meet the first preset requirement, it means that the candidate title does not belong to the category, then the title search for the category can be stopped, and then other categories can be determined in turn in a loop until the remaining titles are divided into corresponding categories. In this way, the category division can be completed for each title. Finally, for each category, the text corresponding to the title under the category is labeled based on the category label of the category, thereby realizing the classification of multiple texts.
[0125] Furthermore, in the above text classification process, no human intervention is required, which is conducive to reducing labor costs and time costs, thereby improving the efficiency of text classification.
[0126] In some implementations, with respect to step S102, when determining the similarity score between any two titles according to the word segmentation result of each of the titles, the following steps may be included:
[0127] (A) According to the plurality of word segmentation results, the plurality of titles are divided into a plurality of title sets, and according to the number of titles included in each of the title sets, a set priority arrangement order between the title sets is determined.
[0128] In this step, it can be understood that multiple titles are divided into units of words to obtain multiple title sets.
[0129] Optionally, when dividing the multiple titles into multiple title sets according to the word segmentation results, the following steps (a) to (b) may be included:
[0130] (a) determining a plurality of groups of target segmented words that meet a second preset requirement from the plurality of segmented word results, and generating a title set for each group of target segmented words based on target titles among the plurality of titles that contain the target segmented words, wherein the plurality of groups of target segmented words are at least partially different from each other.
[0131] It can be understood that the word segmentation result of each title includes multiple word segmentations.
[0132] The target segmentation that meets the second preset requirement may refer to a segmentation that is meaningful to the title and exists in multiple titles at the same time. Here, the number of segmentations included in each group of target segmentations may be one or more, which is not limited here.
[0133] For example, Title 1 is “CEO X of Brand A mobile phone”, and the word segmentation results of Title 1 are: Brand A, mobile phone, CEO, X; Title 2 is “CEO Y of Brand B mobile phone”, and the word segmentation results of Title 2 are: Brand B, mobile phone, CEO, Y; Title 3 is “Brand B mobile phone Y”, and the word segmentation results of Title 3 are: Brand B, mobile phone, Y.
[0134] Then, multiple groups of target segmented words can be determined from the multiple segmented word results. Here, the multiple groups of target segmented words may include B brand, mobile phone, CEO, and Y.
[0135] The generated title set can include the following situations:
[0136] If the target segmentation word is brand B, the corresponding title set 1 is: [CEO of brand B mobile phone Y, brand B mobile phone Y];
[0137] If the target word is CEO, the corresponding title set 2 is: [CEO X of brand A mobile phone, CEO Y of brand B mobile phone];
[0138] If the target segmentation word is Y, the corresponding title set 3 is: [CEO of brand B mobile phone Y, brand B mobile phone Y];
[0139] If the target word is mobile phone, the corresponding title set 4 is: [CEO X of brand A mobile phone, CEO Y of brand B mobile phone, and brand B mobile phone Y].
[0140] (b) obtaining the plurality of title sets based on the title sets corresponding to the plurality of groups of target word segments.
[0141] Here, since each group of target segmented words corresponds to a title set, multiple title sets can be obtained, that is, title set 1, title set 2, title set 3 and title set 4.
[0142] It can be understood that after generating multiple title sets, the set priority arrangement order between the title sets can be determined according to the number of titles included in each title set.
[0143] Here, the priority order of the titles may be determined according to the order of the number of titles.
[0144] Continuing with the above example, the number of titles in title set 1 is 2, the number of titles in title set 2 is 2, the number of titles in title set 3 is 2, and the number of titles in title set 4 is 3, then the order of priority among the title sets can be: title set 4>title set 1, title set 2, title set 3. Here, the priorities among title set 1, title set 2, and title set 3 can be arranged arbitrarily and are not limited here.
[0145] (B) determining a target title set from the plurality of title sets in sequence based on the set priority ranking order, and determining a similarity score between any two titles in the target title set.
[0146] After the set priority ranking order is obtained, a target title set can be determined from the multiple title sets in sequence according to the set priority ranking order, and a similarity score between any two titles in the target title set can be determined.
[0147] Based on the above example, the set priority arrangement order may be: title set 4>title set 1, title set 2, title set 3, then title set 4 may be preferentially determined as the target title set, and the similarity scores between any two titles in title set 4 may be calculated.
[0148] Here, since title set 4 has the largest number of titles, and the similarity scores are calculated separately for each of them, if any other title set (title set 1, title set 2, title set 3) is a subset of title set 4, then there is no need to repeatedly calculate the similarity scores for the titles in other title sets. This can reduce the number of calculations and save resources.
[0149] With respect to step S104, when determining the category label of each category based on the common words contained in each title under each category, the following steps S1041 to S1043 are included:
[0150] S1041: Determine a target category from the multiple categories, and determine a category label of the target category based on common words included in the title corresponding to the target category.
[0151] The number of target categories may be one or more, which is not limited here.
[0152] Optionally, when determining a target category from multiple categories, the multiple categories can be prioritized according to the number of titles corresponding to each category to obtain a category priority arrangement order among the multiple categories, and the target category can be determined from the multiple categories based on the category priority arrangement order.
[0153] Here, the number of titles corresponding to each category can be sorted from large to small to obtain the category priority ranking order among multiple categories, and then the category with a larger category priority can be selected as the target category. For example, the category with the top three category priorities can be selected as the target category.
[0154] Then, based on the common words contained in the title corresponding to the target category, the category label of the target category is determined. Here, if there are multiple target categories, the corresponding category label is determined for each target category.
[0155] Exemplarily, the multiple categories include A, B, C, D, and E, the target categories are A and B, and for target category A, its category labels include A1, A2, and A3, and for target category B, its category labels include B1 and B2.
[0156] S1042: Based on the category label of the target category, label the titles of the other categories among the multiple categories except the target category.
[0157] Here, it can be understood that titles classified into different categories may also contain the same common words. Therefore, after determining the category label of the target category, the titles under other categories can be marked based on the determined category label.
[0158] Continuing with the above example, other categories refer to C, D, and E. Then, based on the determined category labels (i.e., A1, A2, A3, B1, B2), the titles under other categories (C, D, E) can be marked. Specifically, if any title under other categories (C, D, E) contains the above category labels, it will be marked.
[0159] It should be noted here that the marked titles in other categories will not participate in the determination of the category label of the category to which they belong. For example, if title c1 in other category C is marked by A1, then when determining the category label of other category C, it will not be determined based on title c1.
[0160] S1043: For each of the other categories, determine a category label of the other category based on common words included in the unlabeled titles under the other category.
[0161] According to step S1042, since titles in other categories may be marked, the words in these marked titles are not suitable for determining category labels. Therefore, for each other category, the corresponding category label can be determined based on the common words contained in the unmarked titles under the other category.
[0162] Exemplarily, the titles in other categories C include title c1, title c2, title c3, title c4, and title c5, where c1 is labeled by A1. Then, when determining the category label of other categories C, the category label is determined based on the common words contained in titles c2, c3, c4, and c5.
[0163] In some embodiments, for step S1043, when determining the category label of the other category based on the common words contained in the unmarked titles under the other category for each of the other categories, the category label of the other category can be determined based on the common words contained in the unmarked titles under the other category when the number of the unmarked titles under the other category is not less than a first preset number.
[0164] Among them, the first preset number can be set according to actual needs, for example, it can be 3 or 4, and is not limited here.
[0165] Here, if the number of unmarked titles under other categories is not less than the first preset number, it means that the number of unmarked titles is large, and therefore, it is necessary to determine their category labels to complete the subsequent text segmentation.
[0166] On the contrary, if the number of unlabeled titles under other categories is less than the first preset number, it means that the number of these unlabeled titles is small, and therefore, there is no need to determine their category labels.
[0167] See also Figure 3 , is a flowchart of another text classification method provided by an embodiment of the present disclosure. Figure 3 As shown in , the method includes the following steps S301~S307:
[0168] S301: Acquire titles of multiple texts, perform word segmentation processing on each of the titles to obtain word segmentation results, and determine similarity scores between any two titles based on the word segmentation results of each of the titles.
[0169] Among them, step S301 is similar to step S101 in the above embodiment and will not be described in detail here.
[0170] S302: Determine the two first target titles corresponding to the maximum similarity scores as titles in the same category.
[0171] S303: Searching for a candidate title whose sum of similarity scores with the two first target titles is the largest from titles other than the two first target titles.
[0172] Among them, step S302 and step S303 are similar to step S102 in the above embodiment, and are not described in detail here.
[0173] S304: Determine whether the common words between the candidate title and the categorized titles meet the first preset requirement. If yes, execute step S305; if not, execute step S302.
[0174] Here, it is determined whether the common words between the candidate title and the titles that have been divided into categories meet the first preset requirement. If so, it means that the candidate title and the titles that have been divided into categories belong to the same category, so step S305 can be executed. If not, it means that the candidate title and the titles that have been divided into categories do not belong to the same category, and the next category division is performed.
[0175] S305: Classify the candidate titles into categories corresponding to the first target title.
[0176] S306: Determine whether there are any remaining titles, if so, execute step S303, if not, execute step S307.
[0177] Here, if there are remaining titles, it is necessary to further determine whether there are any titles belonging to the same category as the first target title in the remaining titles, so it is necessary to execute step S303 to continue searching for the next candidate title. If there are no remaining titles, it means that all titles have been divided, and then step S307 is executed to determine the category labels of each category.
[0178] S307: Determine a category label for each category based on the common words contained in each title under each category, and perform category labeling on the text corresponding to the title under the category based on the category label.
[0179] Among them, step S307 is similar to step S104 in the above embodiment and will not be described in detail here.
[0180] After classifying the text based on the above embodiment, the classified text can be stored in a text library, and then the target text can be searched from the text library according to actual needs. Based on this, the embodiment of the present disclosure provides a text search method.
[0181] To facilitate understanding of this embodiment, a text search method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the text search method provided in the embodiment of the present disclosure is generally an electronic device, and the electronic device can be a server, which can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data and artificial intelligence platforms. In other implementations, the electronic device can also be a terminal device, wherein the terminal device can be a mobile device, a user terminal, a terminal, a handheld device, a computing device, a vehicle-mounted device, a wearable device, and the like.
[0182] In other implementations, the method can also be applied to an implementation environment consisting of a terminal device and a server. In addition, the text search method can also be implemented by a processor calling a computer-readable instruction stored in a memory.
[0183] See also Figure 4 , is a flowchart of a text search method provided by an embodiment of the present disclosure. Figure 4 As shown in , the text search method includes the following steps S401-S402:
[0184] S401: Acquire a text whitelist; the text whitelist includes at least one preset whitelist word.
[0185] Among them, the preset at least one whitelist word can be set according to actual needs.
[0186] Exemplarily, if the text is an official document, the text whitelist can be set according to department categories. For example, the text whitelist of the environment department can include whitelist words such as air quality and environmental governance, and the text whitelist of the education department can include whitelist words such as entrance examinations. If the text is a medical text, the text whitelist can be set according to medical subjects. For example, the text whitelist of neurosurgery can include whitelist words such as brain tumors, and the text whitelist of dentistry can include whitelist words such as dental health.
[0187] S402: Based on the at least one whitelist word, search for a target text containing the at least one whitelist word from texts stored in a text library; the text library corresponds to a plurality of category labels, each of the category labels corresponds to a plurality of texts, and the plurality of texts corresponding to the category labels are determined based on the text classification method in the aforementioned embodiment.
[0188] It can be understood that some texts in the text library are texts that need to be focused on. Therefore, by obtaining a text whitelist, and then searching for these texts that need to be focused on from the text library based on at least one whitelist word in the text whitelist, the target text is obtained.
[0189] Among them, the text library corresponds to multiple category labels, each category label corresponds to multiple texts, and the multiple texts corresponding to each category label are determined based on the text classification method in the aforementioned embodiment. In this way, it is convenient to manage the text library. Furthermore, by classifying the texts in the text library, when searching based on the text whitelist, targeted searching can be achieved according to the category labels, which is conducive to improving the search efficiency.
[0190] In some implementations, with respect to step S402, when searching for a target text containing the at least one whitelist word from texts stored in a text library based on the at least one whitelist word, the following steps (I) to (III) may be included:
[0191] (I) Obtaining a text blacklist; the text blacklist includes at least one preset blacklist category.
[0192] The text blacklist can be set according to actual needs and is not limited here. Optionally, as the text in the text library continues to increase, the text blacklist can also be updated.
[0193] Here, the at least one blacklist category may refer to a text category that does not require attention.
[0194] (II) Based on the at least one blacklist category, shielding marks are performed on texts in the text library corresponding to the same category label as the at least one blacklist category.
[0195] After obtaining at least one blacklist category, the text corresponding to the category label same as the at least one blacklist category in the text library may be shielded and marked based on the at least one blacklist category, so as to manage the text in the text library.
[0196] Here, the text with a blocking mark may refer to text that will not be processed by default. For example, when a user views text in a text library, the text with a blocking mark will not be displayed. For another example, the text with a blocking mark will not be found by default.
[0197] (III) Based on the at least one whitelist word, searching for the target text from texts in the text library that are not marked as blocked.
[0198] It can be understood that after completing the above-mentioned blocking mark, when searching for the target text, it is possible to search from the text library that is not blocked based on at least one whitelist word, that is, through the blacklist category, the text range and the amount of text can be narrowed, thereby improving the text search efficiency.
[0199] The following is an introduction to the process of generating a text whitelist. Specifically, it includes the following steps (i) to (iii):
[0200] (i) For each category label, determine a target word from the text content of each text corresponding to the category label; the target word refers to the same word whose number in the text is greater than a second preset number.
[0201] The second preset number can be set according to actual needs, and here, the second preset number can be 2. Since the target word refers to the same word whose number in the text is greater than the second preset number, the target word can be understood as a repeated word appearing in the text.
[0202] For example, if the text is: How are you? Really, how are you? Then the target word is "How are you?"
[0203] In some implementations, when determining the target word from the text content of each text, a method without word segmentation may be used, which may specifically include the following steps (A1) to (C1):
[0204] (A1) For each text corresponding to the category label, a position index is constructed for each character in the text, and based on the position index of each character, multiple position indexes corresponding to the same target character are determined.
[0205] The following is an example of a text:
[0206] For example, if the text is "How are you? Really, how are you?", a position index "012345678" is constructed for each character, where each index symbol represents a character, and then multiple position indexes corresponding to the same target character are determined, that is, you [0,5,8], good [1,6], is it [2,7], really [3], of [4],
[0207] (B1) For each target character, any two position indexes of the multiple position indexes of the target character are combined to obtain an index array.
[0208] As can be seen from the foregoing, the target word is actually a repeated word. Therefore, for characters with only a single position index, no subsequent processing is required, and only the target characters need to be processed.
[0209] For example, for the target character "你" (nǐ), it can be determined that the index arrays include [0, 5], [0, 8], and [5, 8]; for the target character "好" (hǎo), the index array can be determined as [1, 6], and for the target character "吗" (ma), the index array can be determined as [2, 7].
[0210] (C1)When the index arrays corresponding to at least two target characters meet the fifth preset requirement, the word composed of the at least two target characters is determined as the target word.
[0211] Here, that the index arrays corresponding to at least two target characters meet the fifth preset requirement means that after adding 1 to the index array corresponding to one target character to obtain a new index array, check whether there is a target character among the remaining target characters whose index array is the same as the new index array. If so, the word composed of the at least two target characters is determined as the target word.
[0212] Continuing with the above example, for the target character "你" (nǐ), its index array includes [0, 5], then [0, 5] + 1 = [1, 6], and the target character corresponding to [1, 6] is "好" (hǎo), that is, "好" (hǎo) exists according to "你" (nǐ). Similarly, [1, 6] + 1 = [2, 7], and the target character corresponding to [2, 7] is "吗" (ma), that is, "吗" (ma) exists according to "好" (hǎo). Further, [2, 7] + 1 = [3, 8], and there is no corresponding target character for [3, 8]. Therefore, "你好吗" (nǐ hǎo ma) can be determined as the target word.
[0213] (ii)Based on the target words corresponding to each text corresponding to the category label, determine the target word set corresponding to the category label.
[0214] By the above steps, the target word in each text can be determined, and thus, according to the target words corresponding to each text, the target word set corresponding to the category label can be determined.
[0215] For example, category label A corresponds to target word set A1, category label B corresponds to target word set B1, and category label C corresponds to target word set C1.
[0216] (iii)Compare the target word sets corresponding to each category label respectively, remove the target words in each target word set that do not meet the fourth preset requirement to obtain a new target word set, and generate the text whitelist based on the target words in the new target word set whose quantity is greater than the third preset quantity.
[0217] Among them, the target words that do not meet the fourth preset requirement may refer to words that are common in each target word set (or multiple target word sets) and have a second frequency. It can be understood that the words that are common in multiple target word sets and have a second frequency are usually meaningless words. Therefore, these words need to be removed to obtain a new target word set.
[0218] Here, the second frequency can be set according to actual needs and is not limited here.
[0219] Furthermore, after obtaining a new target word set, a text whitelist can be generated based on the new target word set. Specifically, target words whose number is greater than a third preset number can be determined from the new target word set as whitelist words through manual search to obtain a text whitelist.
[0220] It can be understood that the target words for manual search are usually some words with practical meanings. At the same time, if their number is greater than the third preset number, it means that the target word is repeated many times and is more important. Therefore, it can be used as a whitelist word. In this way, the accuracy and applicability of the text whitelist can be improved.
[0221] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0222] Based on the same inventive concept, a text classification device corresponding to the above-mentioned text classification method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned text classification method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0223] See also Figure 5 , Figure 5 Schematic diagram of a text classification device provided by an embodiment of the present disclosure. Figure 5 As shown in , the text classification device 500 provided by the embodiment of the present disclosure includes:
[0224] The text title acquisition module 510 is used to acquire titles of multiple texts, perform word segmentation processing on each of the titles, obtain word segmentation results, and determine the similarity score between any two titles based on the word segmentation results of each of the titles;
[0225] The first title division module 520 is used to determine the two first target titles corresponding to the maximum similarity scores as titles under the same category, and to search for a candidate title with the maximum sum of similarity scores with the two first target titles from other titles except the two first target titles, and to classify the candidate title into the category corresponding to the first target title when the common words between the candidate title and the titles that have been divided into categories meet the first preset requirement, and to return to the step of searching for the candidate title with the maximum sum of similarity scores with the two first target titles until the common words between the candidate title and the titles that have been divided into categories do not meet the first preset requirement, and the search is stopped;
[0226] The second title classification module 530 is used for returning to the step of determining the two first target titles corresponding to the maximum similarity scores as titles under the same category for the remaining titles among the multiple titles except the titles that have been classified into categories, until the remaining titles are respectively classified into corresponding categories;
[0227] The category label determination module 540 is used to determine the category label of each category based on the common words contained in each title under each category, and to perform category labeling on the text corresponding to the title under the category based on the category label.
[0228] In a possible implementation, the text title acquisition module 510 is specifically used to:
[0229] According to the plurality of word segmentation results, the plurality of titles are divided into a plurality of title sets, and according to the number of titles included in each of the title sets, a set priority arrangement order between the title sets is determined;
[0230] A target title set is determined from the plurality of title sets in sequence based on the set priority ranking order, and a similarity score between any two titles in the target title set is determined.
[0231] In a possible implementation, the text title acquisition module 510 is specifically used to:
[0232] Determining multiple groups of target segmented words that meet the second preset requirement from the multiple segmented word results, and generating a title set for each group of target segmented words based on target titles containing the target segmented words among the multiple titles, wherein the multiple groups of target segmented words are at least partially different from each other;
[0233] Based on the title sets corresponding to the multiple groups of target word segments, the multiple title sets are obtained.
[0234] In a possible implementation, the text title acquisition module 510 is specifically used to:
[0235] Perform word segmentation processing on each of the titles to obtain an initial word segmentation result;
[0236] The word segmentation results are obtained by removing the word segmentations that do not meet the third preset requirement in the initial word segmentation results.
[0237] In a possible implementation, the category label determination module 540 is specifically configured to:
[0238] Determining a target category from the plurality of categories, and determining a category label of the target category based on common words included in the titles corresponding to the target category;
[0239] Based on the category label of the target category, marking titles under other categories among the multiple categories except the target category;
[0240] For each of the other categories, a category label of the other category is determined based on common words contained in unlabeled titles under the other category.
[0241] In a possible implementation, the category label determination module 540 is specifically configured to:
[0242] According to the number of titles corresponding to each of the categories, the multiple categories are prioritized to obtain a category priority arrangement order among the multiple categories, and according to the category priority arrangement order, the target category is determined from the multiple categories.
[0243] In a possible implementation, the category label determination module 540 is specifically configured to:
[0244] For each of the other categories, when the number of unlabeled titles under the other categories is not less than a first preset number, the category label of the other category is determined based on the common words contained in the unlabeled titles under the other categories.
[0245] See also Figure 6 , Figure 6 Schematic diagram of a text search device provided by an embodiment of the present disclosure. Figure 6 As shown in , the text search device 600 provided by the embodiment of the present disclosure includes:
[0246] The acquisition module 610 is used to acquire a text whitelist; the text whitelist includes at least one preset whitelist word;
[0247] The search module 620 is used to search for a target text containing the at least one whitelist word from the text stored in the text library based on the at least one whitelist word; the text library corresponds to multiple category labels, each of the category labels corresponds to multiple texts, and the multiple texts corresponding to each category label are determined based on the text classification method described in any of the aforementioned embodiments.
[0248] In a possible implementation manner, the search module 620 is specifically configured to:
[0249] Acquire a text blacklist; the text blacklist includes at least one preset blacklist category;
[0250] Based on the at least one blacklist category, shielding and marking texts corresponding to the same category label as the at least one blacklist category in the text library;
[0251] Based on the at least one whitelist word, the target text is searched from texts in the text library that are not marked with a blocking mark.
[0252] See also Figure 7 , Figure 7 Schematic diagram of another text search device provided by an embodiment of the present disclosure. Figure 7 As shown in , the text search device 600 further includes a generating module 630, and the generating module 630 is used to:
[0253] For each category label, determining a target word from the text content of each text corresponding to the category label; the target word refers to the same word whose number of words in the text is greater than a second preset number;
[0254] Determine a target word set corresponding to the category label based on the target words corresponding to each text corresponding to the category label;
[0255] The target word sets corresponding to each of the category labels are compared respectively, and the target words in each of the target word sets that do not meet the fourth preset requirements are removed to obtain a new target word set, and based on the target words in the new target word set whose number is greater than the third preset number, the text whitelist is generated.
[0256] In a possible implementation, the generating module 630 is specifically configured to:
[0257] For each text corresponding to the category label, construct a position index for each character in the text, and based on the position index of each character, determine multiple position indexes corresponding to the same target character;
[0258] For each target character, any two position indexes of the target character are combined to obtain an index array;
[0259] When the index arrays respectively corresponding to at least two target characters meet the fifth preset requirement, the word composed of the at least two target characters is determined as the target word.
[0260] For descriptions of the processing flow of each module in the above-mentioned device and the interaction flow between each module, reference can be made to the relevant descriptions in the above-mentioned method embodiment, which will not be described in detail here.
[0261] Corresponding to the above-mentioned text classification method or text search method, the embodiment of the present disclosure further provides an electronic device 800, such as Figure 8 FIG. 8 is a schematic diagram of the structure of an electronic device 800 provided in an embodiment of the present disclosure, including:
[0262] Processor 810, memory 820, and bus 830. The memory 820 is used to store execution instructions, including internal memory 821 and external memory 822; the internal memory 821 is also called internal memory, which is used to temporarily store the operation data in the processor 810 and the data exchanged with the external memory 822 such as a hard disk. The processor 810 exchanges data with the external memory 822 through the internal memory 821.
[0263] In the embodiment of the present application, the memory 820 is specifically used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 810. That is, when the electronic device 800 is running, the processor 810 communicates with the memory 820 through the bus 830, so that the processor 810 executes the application code stored in the memory 820, and then executes the text classification method or text search method described in the above method embodiment.
[0264] Among them, the memory 820 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.
[0265] The processor 810 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0266] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 800. In other embodiments of the present application, the electronic device 800 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0267] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any of the text classification methods or text search methods described in the above method embodiments is executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.
[0268] The method in the embodiments of the present disclosure may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM or other programmable device.
[0269] The computer program or instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.
[0270] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A text classification method, characterized in that: include: Obtaining titles of multiple texts, performing word segmentation processing on each of the titles to obtain word segmentation results, and determining similarity scores between any two titles based on the word segmentation results of each of the titles; Determine the two first target titles corresponding to the maximum similarity scores as titles under the same category, search for a candidate title with the maximum sum of similarity scores with the two first target titles from other titles except the two first target titles, and classify the candidate title into the category corresponding to the first target title if the common words between the candidate title and the titles that have been classified meet the first preset requirement, return to the step of searching for the candidate title with the maximum sum of similarity scores with the two first target titles, and stop searching until the common words between the candidate title and the titles that have been classified do not meet the first preset requirement; For the remaining titles except the titles that have been divided into categories, return to the step of determining the two first target titles corresponding to the maximum similarity scores as titles under the same category until the remaining titles are respectively divided into corresponding categories; Based on the common words contained in each title under each category, a category label of each category is determined, and based on the category label, the text corresponding to the title under the category is categorized.
2. The method according to claim 1, characterized in that Determining the similarity score between any two titles according to the word segmentation result of each title includes: According to the plurality of word segmentation results, the plurality of titles are divided into a plurality of title sets, and according to the number of titles included in each of the title sets, a set priority arrangement order between the title sets is determined; A target title set is determined from the plurality of title sets in sequence based on the set priority ranking order, and a similarity score between any two titles in the target title set is determined.
3. The method according to claim 2, characterized in that The step of dividing the plurality of titles into a plurality of title sets according to the plurality of word segmentation results comprises: Determining multiple groups of target segmented words that meet the second preset requirement from the multiple segmented word results, and generating a title set for each group of target segmented words based on target titles containing the target segmented words among the multiple titles, wherein the multiple groups of target segmented words are at least partially different from each other; Based on the title sets corresponding to the multiple groups of target word segments, the multiple title sets are obtained.
4. The method according to claim 1, characterized in that: The word segmentation of each of the titles to obtain the word segmentation results includes: Perform word segmentation processing on each of the titles to obtain an initial word segmentation result; The word segmentation results are obtained by removing the word segmentations that do not meet the third preset requirement in the initial word segmentation results.
5. The method according to claim 1, characterized in that Determining a category label for each category based on the common words contained in each title under each category includes: Determining a target category from the plurality of categories, and determining a category label of the target category based on common words included in the titles corresponding to the target category; Based on the category label of the target category, marking titles under other categories among the multiple categories except the target category; For each of the other categories, a category label of the other category is determined based on common words contained in unlabeled titles under the other category.
6. The method according to claim 5, characterized in that Determining a target category from a plurality of categories comprises: According to the number of titles corresponding to each of the categories, the multiple categories are prioritized to obtain a category priority arrangement order among the multiple categories, and according to the category priority arrangement order, the target category is determined from the multiple categories.
7. The method according to claim 5, characterized in that The step of determining, for each of the other categories, a category label of the other category based on common words contained in unlabeled titles under the other category, includes: For each of the other categories, when the number of unlabeled titles under the other categories is not less than a first preset number, the category label of the other category is determined based on the common words contained in the unlabeled titles under the other categories.
8. A text search method, characterized in that: include: Get the text whitelist; The text whitelist includes at least one preset whitelist word; Based on the at least one whitelist word, searching for a target text containing the at least one whitelist word from texts stored in a text library; The text library corresponds to multiple category labels, each of the category labels corresponds to multiple texts, and the multiple texts corresponding to each of the category labels are determined based on the text classification method described in any one of claims 1-7.
9. The method according to claim 8, characterized in that The step of searching, based on the at least one whitelist word, for a target text containing the at least one whitelist word from texts stored in a text library comprises: Acquire a text blacklist; the text blacklist includes at least one preset blacklist category; Based on the at least one blacklist category, shielding and marking texts corresponding to the same category label as the at least one blacklist category in the text library; Based on the at least one whitelist word, the target text is searched from texts in the text library that are not marked with a blocking mark.
10. The method according to claim 8 or 9, characterized in that: The text whitelist is generated by the following method: For each category label, determining a target word from the text content of each text corresponding to the category label; the target word refers to the same word whose number of words in the text is greater than a second preset number; Determine a target word set corresponding to the category label based on the target words corresponding to each text corresponding to the category label; The target word sets corresponding to each of the category labels are compared respectively, and the target words in each of the target word sets that do not meet the fourth preset requirements are removed to obtain a new target word set, and based on the target words in the new target word set whose number is greater than the third preset number, the text whitelist is generated.
11. The method according to claim 10, characterized in that The step of determining the target word from the text content of each text corresponding to the category label includes: For each text corresponding to the category label, construct a position index for each character in the text, and based on the position index of each character, determine multiple position indexes corresponding to the same target character; For each target character, any two position indexes of the target character are combined to obtain an index array; When the index arrays respectively corresponding to at least two target characters meet the fifth preset requirement, the word composed of the at least two target characters is determined as the target word.
12. A text classification device, characterized in that: include: A text title acquisition module is used to acquire titles of multiple texts, perform word segmentation processing on each of the titles, obtain word segmentation results, and determine the similarity score between any two titles based on the word segmentation results of each of the titles; A first title classification module is used to determine the two first target titles corresponding to the maximum similarity scores as titles under the same category, and to search for a candidate title with the maximum sum of similarity scores with the two first target titles from other titles other than the two first target titles, and to classify the candidate title into the category corresponding to the first target title if the common words between the candidate title and the titles that have been classified meet the first preset requirement, and to return to the step of searching for the candidate title with the maximum sum of similarity scores with the two first target titles until the common words between the candidate title and the titles that have been classified do not meet the first preset requirement, and the search is stopped; A second title division module is used for returning to the step of determining the two first target titles corresponding to the maximum similarity scores as titles under the same category for the remaining titles among the multiple titles except the titles that have been divided into categories, until the remaining titles are respectively divided into corresponding categories; The category label determination module is used to determine the category label of each category based on the common words contained in each title under each category, and to perform category marking on the text corresponding to the title under the category based on the category label.
13. A text search device, characterized in that: include: Acquisition module, used to obtain text whitelist; The text whitelist includes at least one preset whitelist word; A search module, configured to search for a target text containing the at least one whitelist word from texts stored in a text library based on the at least one whitelist word; The text library corresponds to multiple category labels, each of the category labels corresponds to multiple texts, and the multiple texts corresponding to each of the category labels are determined based on the text classification method according to any one of claims 1 to 7.
14. An electronic device, characterized in that: The electronic device comprises a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the text classification method as described in any one of claims 1 to 7 or the text search method as described in any one of claims 8 to 11 is executed.
15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the text classification method according to any one of claims 1 to 7 or the text search method according to any one of claims 8 to 11.
Citation Information
Patent Citations
Title classification method and device
CN107729917A
Text classification method and device, terminal equipment and storage medium
CN110851598A
Text processing method and device, computer equipment and storage medium
CN112560444A
Text category determination method and device, computer equipment and storage medium
CN115129871A
Generation of classification data used for classifying documents
US20170351688A1
