A text data classification method, device, equipment and storage medium

Abstract

The present application relates to the field of data processing and the field of digital medical treatment, and in particular to a text data classification method, device and equipment and storage medium, which can be applied to the classification of text data of insurance products in the medical field. By obtaining a text data set to be classified, a set of synonyms corresponding to a keyword is extracted, a text data keyword node is constructed based on the mapping relationship between the pre-labeled text data and the keyword and the set of synonyms corresponding to the keyword, and the set is searched. The connected blocks corresponding to the text data nodes in the set are traversed, the keyword nodes are grouped according to the association relationship between each node and the root node, the keyword category of the text data is determined according to the grouping result, the obtained keyword is classified and stored in the database. The connected blocks of the union set are matched by using synonyms, which reduces the time complexity and space complexity of the classification process, improves the processing efficiency of the text data, and ensures that the user matches the corresponding medical insurance product.

Description

Technical Field

[0001] This invention relates to the fields of data processing and digital healthcare, and particularly to a method, apparatus, device, and storage medium for classifying text data. Background Technology

[0002] With the rapid development of the internet, internet-based healthcare has been widely accepted by the public, supporting functions such as disease-aided diagnosis, health management, and remote consultations. Doctors conducting online consultations and prescribing medications is gradually becoming a trend. Simultaneously, as public awareness of health insurance and critical illness insurance increases, commercial medical insurance is becoming an important supplement to social medical insurance. Linking individual health and illness risks with commercial medical insurance can effectively alleviate the predicament of insufficient medical funds in the event of a sudden serious illness. Therefore, it is necessary to categorize the textual data of insurance products in the medical field to match users with corresponding insurance products based on the type of textual data.

[0003] In existing technologies, most methods for classifying text data related to insurance products in the medical field combine DBSCAN and TF-IDF. This involves using a word segmentation tool (jieba) to segment the title of a text dataset, then employing the TF-IDF algorithm to extract keywords and convert them into vectors. Finally, a K-means clustering algorithm is used to form K clusters based on similar content. However, the DBSCAN algorithm requires a cluster radius and a minimum number of clusters. Determining the correct cluster radius is challenging; a large radius may group dissimilar text data into a single cluster, while a small radius may separate similar text data into two separate clusters. Therefore, improving the efficiency of insurance product text data classification and optimizing the time and space complexity of the entire classification process has become a pressing technical problem for those skilled in the art. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, device, medium, and product for classifying text data to address the aforementioned technical problems, thereby solving the problem of excessive time and space complexity in the existing text data classification process and making it easier to classify text data of insurance products.

[0005] A first aspect of this application provides a method for classifying text data, the method comprising:

[0006] Obtain the text dataset to be classified, and extract the set of synonyms corresponding to the keywords in the text dataset to be classified;

[0007] Based on the pre-defined mapping relationship between text data and keywords, and the set of synonyms corresponding to the keywords, a text data keyword node union-find set is constructed;

[0008] Traverse the connected components corresponding to the text data nodes in the union set, and group the keyword nodes in the union set according to the association between each node and the root node, so as to determine the keyword category of the text data based on the grouping results.

[0009] Based on the keyword categories of the text data, the obtained keywords are categorized and stored in the database.

[0010] A second aspect of this application provides a text data classification device, the text data classification device comprising:

[0011] The acquisition module is used to acquire the text dataset to be classified and extract the set of synonyms corresponding to the keywords in the text dataset to be classified.

[0012] The construction module is used to construct text data keyword nodes and perform a lookup based on the pre-defined mapping relationship between text data and keywords and the set of synonyms corresponding to the keywords;

[0013] The grouping module is used to traverse the text data nodes and look up the corresponding connected components in the union set, and group the keyword nodes in the union set according to the association relationship between each node and the root node, so as to determine the keyword category of the text data based on the grouping results.

[0014] The storage module is used to categorize and store the obtained keywords into the database based on the keyword categories of the text data.

[0015] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the text data classification method as described in the first aspect.

[0016] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the text data classification method as described in the first aspect.

[0017] In summary, this invention provides a method, apparatus, device, and storage medium for classifying text data, applicable to the classification of text data related to insurance products in the medical field. By acquiring a text dataset to be classified, a set of synonyms corresponding to the keywords in the dataset is extracted. Based on a pre-defined mapping relationship between text data and keywords, and the set of synonyms corresponding to the keywords, a disjoint-set data structure (DFS) for the text data keyword nodes is constructed. Then, the connected components corresponding to the DFS are traversed, and the keyword nodes in the DFS are grouped according to the association relationship between each node and the root node. The keyword category of the text data is determined based on the grouping results, and the obtained keywords are classified and stored in a database. Therefore, this application uses synonyms to match the connected components of the DFS, thereby reducing the time and space complexity of the classification process, eliminating the need for repeated calculations. Furthermore, by classifying and storing the obtained keywords in the database according to the keyword category of the text data, the type of the next piece of text data is directly determined, improving the processing efficiency of text data keywords, enhancing the user experience, and ensuring that users are matched with the corresponding medical insurance products. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an application environment for a text data classification method provided in an embodiment of the present invention;

[0020] Figure 2 This is a flowchart illustrating a text data classification method provided in an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the structure of a text data classification device provided in an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0027] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0029] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0030] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0031] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0032] An embodiment of the present invention provides a method for classifying text data, which can be applied to, for example... Figure 1 In this application environment, the client and server communicate with each other. Clients include, but are not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The server can be implemented using a standalone server or a server cluster, through which medical data, such as personal health records, prescriptions, and examination reports, can be uploaded and downloaded.

[0033] See Figure 2 This is a flowchart illustrating a text data classification method provided in an embodiment of the present invention. The text data classification method described above can be applied to... Figure 1 The server in the above-mentioned configuration connects to the corresponding client, such as... Figure 2 As shown, the method for classifying this text data may include the following steps.

[0034] S201: Obtain the text dataset to be classified, and extract the set of synonyms corresponding to the keywords in the text dataset to be classified.

[0035] In step S201, the text data classification method provided in this application embodiment can be applied to Natural Language Processing (NLP) scenarios, specifically in the medical field. Blockchain can be used to store medical text data, such as personal health records, prescriptions, and examination reports. The business system can obtain the text dataset of insurance products to be classified through external input or online acquisition. The business system can extract keywords from the text dataset of insurance products to be classified and create corresponding synonym sets based on the keywords. This business system can be a core system for storing medical insurance product information; all medical insurance product information within the enterprise is stored in this core system. Different medical insurance product information documents are generated and stored in this core system according to different coverage plans. These medical insurance product information documents store information such as product name, commodity code, product code, insurance type code, coverage amount, coverage period, payment period, premium, product introduction, insurance type, assessment rules, underwriting rules, acceptance rules, and policy signing rules. The associated system may be a publishing system for publishing medical insurance products to third-party e-commerce platforms, such as a programming system used to edit program code to publish medical insurance products.

[0036] In one possible implementation, the data is medical data, such as personal health records, prescriptions, examination reports, etc.

[0037] In one possible implementation, the natural language text is medical text data, which may be an electronic healthcare record, an electronic personal health record, including medical records, electrocardiograms, medical images, and other electronic records with value for preservation and future reference.

[0038] Information retrieval and keyword extraction have become channels for users to quickly obtain the information they need in many scenarios. For example, in the medical field, artificial intelligence models can be used to query massive amounts of electronic medical records or extract the medical record information needed by users through keywords, which helps to provide users with medical record references and thus ensures the security of medical text data.

[0039] Furthermore, by extracting keywords from the text data, a set of synonyms corresponding to the keywords is obtained. These keywords are words that embody the main theme of the text data, appearing either in the title or in the first and last paragraphs. Keyword extraction involves identifying the title of the text data and obtaining the words contained within it, using these words as the keywords. Specifically, the keywords are first matched with words pre-stored in a database, and the nature of the keywords is determined based on the properties of the matched words. The properties of the words stored in the database have been pre-defined. The set of synonyms could be sports-fitness, tourism-travel, food-foodie, insurance-insurance, disease-medication, etc., but this embodiment does not impose specific limitations.

[0040] Optionally, extract the set of synonyms corresponding to the keywords in the text dataset to be classified, including:

[0041] Feature extraction is performed on the keywords to obtain the keyword features;

[0042] Determine whether the similarity of keyword features between any two keywords is less than a preset similarity threshold;

[0043] If the similarity of the keyword features of any two keywords is less than a preset similarity threshold, then these two keywords are identified as synonyms. This process continues until all keywords in the text data are identified, thereby determining the set of synonyms corresponding to the keywords in the categorized text dataset.

[0044] Specifically, by extracting keyword features from the text dataset to be classified, the similarity between any two keywords is determined based on these features. If the similarity is less than a preset similarity threshold, the two keywords are considered synonyms. This process is repeated for all keywords in the text dataset until all keywords are classified, thus determining the set of synonyms for each keyword in the dataset. If the similarity is not less than the preset similarity threshold, the process ends. In other words, if the similarity is not less than the preset similarity threshold, subsequent steps in the text classification method will not be executed. Constructing the synonym set requires manual annotation. For example, the synonym pair "sports-fitness" will be stored, and another synonym pair "fitness-bodybuilding" can be linked using a disjoint-set data structure. For words that are difficult to judge manually, we need to add Embedding technology to determine synonyms. This involves using the word2vec algorithm to calculate the vectors of two words and the distance between them. If the distance is less than a set threshold t, the two words are considered synonyms.

[0045] In this embodiment, a set of synonyms corresponding to the keywords in the text data dataset to be classified is extracted so that the connected component matching of the disjoint set can be achieved using synonyms in subsequent work, thereby enabling faster classification of the text data.

[0046] S202: Construct a disjoint-set data structure for text data keyword nodes based on the pre-defined mapping relationship between text data and keywords, as well as the set of synonyms corresponding to the keywords.

[0047] In step S202, this application constructs text data keyword nodes and performs lookup by combining the pre-defined mapping relationship between text data and keywords with the set of synonyms corresponding to the keywords.

[0048] Optionally, construct a disjoint-set data structure for the text data keyword nodes, including:

[0049] Create a full content table of the text data, which includes keywords, time, identifiers, and storage location;

[0050] Based on the pre-defined mapping relationship between text data and keywords, determine whether the keywords in the full content table of the text data are consistent with the set of synonyms corresponding to the keywords;

[0051] If the keywords in the full content table of the text data are consistent with the set of synonyms corresponding to the keywords, then the corresponding keywords are extracted, and a text data keyword node is constructed and a lookup set is performed.

[0052] It is understood that this application creates a full content table for text data. This full content table is a table of all published articles stored in a Hive database, including keywords, time, content, title, and storage location. The full content table for text data is diverse. Taking the medical field as an example, the full content table for text data may include information corresponding to keywords such as product name, price, gender, coverage plan, coverage age, coverage period, and quantity for medical insurance. Based on the pre-defined mapping relationship between text data and keywords, it is determined whether the keywords in the full content table of text data match the set of synonyms corresponding to those keywords. If the keywords in the full content table of text data match the set of synonyms corresponding to those keywords, the corresponding keywords are extracted, and a keyword node for text data is constructed and searched. If the keywords in the full content table of text data do not match the set of synonyms corresponding to those keywords, the process ends, meaning that the subsequent steps of the text data classification method will not be executed. For the full content table, each piece of text data may have at most two keywords. First, the keyword list vector for each item corresponding to each piece of text data is extracted: [keyword1, keyword2]. Here, "keyword" represents the central theme or keywords of the text data, such as "sports" and "fitness." The built-in Python word segmentation library, jieba, is used to extract these keywords, creating a mapping between text data items and vector lists—item:vector1. For example, if the vector corresponding to text data item1 contains keywords like "sports" and "fitness," then the storage format would be—item1: "sports" and "fitness." For instance, jieba is used to extract words from the title of text data. For example, if the title of an article is "Badminton is a great fitness activity," the keywords extracted by jieba would be "badminton, sports, fitness."

[0053] In this embodiment, a text data keyword node and lookup set is constructed based on the pre-labeled mapping relationship between text data and keywords and the set of synonyms corresponding to the keywords. This makes keyword extraction more accurate and avoids complex deep learning models, thus laying a good foundation for subsequent keyword processing and classification of text data according to keyword type.

[0054] S203: Traverse the connected components corresponding to the text data nodes in the union-find set, and group the keyword nodes in the text data keyword node union-find set according to the association relationship between each node and the root node, so as to determine the keyword category of the text data based on the grouping results.

[0055] In step S203, after creating the union-find, the system traverses the connected components corresponding to the text data nodes in the union-find. Based on the association between each keyword node and the root node in the union-find, the keyword nodes in the text data keyword node union-find are grouped to determine the keyword category of the text data. The corresponding union-find is an algorithm for calculating connectivity. Initially, each node points to itself. If node u and node v are connected, they must have the same root node. This process is completed by the find function, whose main function is to recursively traverse from a given node to the root. For example, the synonyms "sports" and "fitness" belong to the same connected component after being added to the union-find and point to the same root node. When the keyword "sports" or "fitness" appears in an article, the find function can be used to find the corresponding root node. A hash table is used to record the articles corresponding to that root node to complete the classification. The connected component contains all synonyms, and the node of each word points to the same root node.

[0056] Optionally, before traversing the text data nodes and finding the corresponding connected components in the set, the following steps are included:

[0057] Initialize the text data keyword nodes and perform a lookup;

[0058] The first keyword node and the second keyword node are determined from the text data keyword node union set, wherein the first keyword node and the second keyword node are two text data keyword topology nodes connected by an edge;

[0059] Search for the root nodes of the first keyword node and the second keyword node, and merge the root nodes of the first keyword node and the second keyword node.

[0060] Traverse all node connections in the text data keyword node union set and determine the root node merging result for all first and second nodes.

[0061] In this embodiment, by initializing the keyword nodes in the text data disjoint-set data structure, and based on the keyword topology nodes in the disjoint-set data structure, the root node of each keyword topology node is set to the keyword topology node itself. The keyword topology nodes are read, and the node_id is stored in the dictionary set Grid_dus and initialized so that Grid_dus[node_id] = node_id. That is, during initialization, the root node of all nodes in the keyword is the node itself. If the total number of nodes is N, then after initialization, the number of root nodes in Grid_dus is also N. For an edge e connecting nodes m and n in the keyword, the root node pointed to by m and n is found and denoted as r. m and r n Let Grid_dus[r m ] = r n That is, by using the connection relationship of edge 'e' in the keyword, the root node of node m is pointed to the root node of node n. Then, for any starting node in the keyword node disjoint-setup, the root node search process is as follows: check if the root node value corresponding to the starting node is equal to the starting node. If yes, then the starting node is its own root node. If not, then update the root node corresponding to the starting node to the starting node, and then continue the search. In this way, all node connection relationships in the keyword node disjoint-setup of the text data are traversed to determine the merged root node results of all first and second nodes.

[0062] Optionally, the keyword nodes in the text data node union-find set are grouped according to the association relationship between each node and the root node, including:

[0063] Traverse the connected components corresponding to the text data keyword nodes in the union set, and determine whether the current keyword is consistent with the connected components corresponding to the text data keyword nodes in the union set according to the association relationship between each node and the root node in the union set.

[0064] If the current keyword matches the connected component in the disjoint-set data structure of the text data keyword nodes, then the keyword nodes in the disjoint-set data structure of the text data nodes are grouped to obtain the grouping results.

[0065] Specifically, by traversing the connected components corresponding to the disjoint-set data structure of text data keyword nodes, and based on the association between each node in the disjoint-set data structure and the root node, it is determined whether the current keyword matches the corresponding connected component in the disjoint-set data structure. If the current keyword matches the corresponding connected component, the keyword nodes in the disjoint-set data structure are grouped to obtain the grouping results. If the current keyword does not match the corresponding connected component, the process ends, meaning that the subsequent steps of the text data classification method will not be executed. That is, for the two keywords corresponding to the text data item1, the corresponding connected components are searched in the disjoint-set data structure. The root node is found using the find function, indicating that the keyword "sports" or "fitness" belongs to the same connected component, and that text data item1 has found a category. At this point, the article item1 is added to the list corresponding to the root node using a hash table: map[root].append(item1). In this way, after the disjoint-set data structure is executed, it is clear which articles belong to the same category.

[0066] For example, the process of calculating the disjoint-set connected component in this application is as follows:

[0067] 1. Number all known words (assuming there are n words) from 1 to n. At the same time, create a disjoint set array parent of size n, where each position stores a word. Make the parent node of each element point to itself, i.e., parent[i] = i.

[0068] 2. If two words are synonyms and their indices are i and j respectively, the union function can be used to connect i and j, i.e., parent[find(i)] = find(j). This will make the parent nodes of the two indices point to the same root node, and at the same time make i and j belong to the same connected component.

[0069] It should be noted that, finally, the text data nodes are traversed and the corresponding connected components are searched in the set. For each root node, all nodes pointing to the root node are grouped together, and each group corresponds to a keyword. In this way, the keyword grouping result of the text data can be obtained.

[0070] Optionally, the keyword categories of the text data can be determined based on the grouping results, including:

[0071] Based on the grouping results, determine the regional characteristics and weights of the keywords in the text data;

[0072] Based on the regional characteristics and weights of the keywords, the keyword categories of the text data are determined.

[0073] Specifically, by grouping keywords, the regional characteristics carried by the keywords and their weights within the text data are determined. Regional characteristics refer to the semantic information that keywords in the text data can represent regionality, such as the location where the text data was published, place names within the text data, or the location of the publishing entity. For example, the publishing entity could be a local insurance company or a local television station. The publishing entity of the text data may or may not be regional. For instance, when the publishing entity is a local insurance company, which typically only reports on local events, the publishing entity of the text data is regional, meaning the text data is a regional article. When the publishing entity is a national insurance company, which typically reports on national events, the text data is not regional, meaning the text data is a non-regional article. Weight refers to the proportion of the keyword in the text data, such as 20%, 60%, or 90%, etc. This application does not impose any limitations. By considering the regional characteristics and weights of the keywords, the keyword category of the text data is determined.

[0074] In this embodiment, the keyword nodes in the disjoint-set data structure are grouped by disjoint-set data structure to determine the keyword category of the text data based on the grouping results. This ensures that the time complexity of the disjoint-set data structure is reduced to O(N*α(N)). Without complex deep learning models, the disjoint-set data structure is simpler to implement, thereby improving the processing efficiency of text data keywords and reducing system complexity and space complexity.

[0075] S204: Based on the keyword categories of the text data, the obtained keywords are categorized and stored in the database.

[0076] In step S204, after determining the keyword categories of the text data, the obtained keywords are categorized and stored in the database. In this way, when a new piece of text data arrives, it can be directly searched by keywords and the connected components in the set can be retrieved to find the category to which the article belongs, without the need for full and repeated calculations.

[0077] Optionally, the obtained keywords are categorized and stored in the database according to the keyword categories of the text data, including:

[0078] Based on the keyword categories of the text data, determine whether the keyword categories of the next text data belong to keyword categories that violate preset rules;

[0079] If the keyword category of the next text data belongs to a keyword category that violates the preset rules, then the text data of the next text will be blocked.

[0080] If the keyword category of the next text data does not belong to the keyword category that violates the preset rules, the obtained keywords will be categorized and stored in the database, and the type of the next text data will be directly determined.

[0081] Specifically, based on the keyword categories of the text data, it is determined whether the keyword categories of the next text data belong to the category that violates preset rules, that is, whether the keyword categories of the next text data belong to the category that deviates from the purpose of medical insurance. If the keyword categories of the next text data belong to the category that deviates from the purpose of medical insurance, then the keyword categories of the next text data are determined to be articles that violate preset rules, and the keywords of the next text data are marked as articles that deviate from the purpose of medical insurance, thereby blocking the next text data. If the keyword categories of the next text data do not belong to the category that violates preset rules, then the obtained keywords are classified and stored in the database, and the type of the next text data is directly determined. In this way, the new text data can be directly searched by keywords and the connected components in the set can be found to find the article's type, without having to perform full and repeated calculations.

[0082] In this embodiment, keywords are categorized and stored in the database based on the keyword categories of the text data. There is no need to use K-means and DBSCAN for clustering and perform full repeated calculations. Only the results of the disjoint set calculation can be saved. New text data can use the find function to find the root node in the saved keywords, which greatly optimizes the time and space complexity of the entire classification process and improves the accuracy of text data classification.

[0083] In summary, this invention provides a method, apparatus, device, and storage medium for classifying text data, applicable to the classification of text data related to insurance products in the medical field. By acquiring a text dataset to be classified, a set of synonyms corresponding to the keywords in the dataset is extracted. Based on a pre-defined mapping relationship between text data and keywords, and the set of synonyms corresponding to the keywords, a disjoint-set data structure (DFS) for the text data keyword nodes is constructed. Then, the connected components corresponding to the DFS are traversed, and the keyword nodes in the DFS are grouped according to the association relationship between each node and the root node. The keyword category of the text data is determined based on the grouping results, and the obtained keywords are classified and stored in a database. Therefore, this application uses synonyms to match the connected components of the DFS, thereby reducing the time and space complexity of the classification process, eliminating the need for repeated calculations. Furthermore, by classifying and storing the obtained keywords in the database according to the keyword category of the text data, the type of the next piece of text data is directly determined, improving the processing efficiency of text data keywords, enhancing the user experience, and ensuring that users are matched with the corresponding medical insurance products.

[0084] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of the text data classification device provided in an embodiment of the present invention. In this embodiment, the terminal includes units used for performing... Figure 2 The steps in the corresponding embodiments. Please refer to the details. Figure 2 as well as Figure 2 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 3 The text data classification device 30 includes: an acquisition module 31, a construction module 32, a grouping module 33, and a storage module 34.

[0085] The acquisition module 31 is used to acquire the text dataset to be classified and extract the set of synonyms corresponding to the keywords in the text dataset to be classified.

[0086] Construction module 32 is used to construct text data keyword nodes and lookup set based on the pre-defined mapping relationship between text data and keywords and the set of synonyms corresponding to the keywords;

[0087] Grouping module 33 is used to traverse the connected components in the text data node union set and group the keyword nodes in the text data keyword node union set according to the association relationship between each node and the root node, so as to determine the keyword category of the text data based on the grouping results.

[0088] The storage module 34 is used to categorize and store the obtained keywords into the database according to the keyword categories of the text data.

[0089] Optionally, the acquisition module 31 described above is specifically used for:

[0090] Feature extraction is performed on the keywords to obtain the keyword features;

[0091] Determine whether the similarity of keyword features between any two keywords is less than a preset similarity threshold;

[0092] If the similarity of the keyword features of any two keywords is less than a preset similarity threshold, then these two keywords are identified as synonyms. This process continues until all keywords in the text data are identified, thereby determining the set of synonyms corresponding to the keywords in the categorized text dataset.

[0093] Optionally, the aforementioned building module 32 is specifically used for:

[0094] Create a full content table of the text data, which includes keywords, time, identifiers, and storage location;

[0095] Based on the pre-defined mapping relationship between text data and keywords, determine whether the keywords in the full content table of the text data are consistent with the set of synonyms corresponding to the keywords;

[0096] If the keywords in the full content table of the text data are consistent with the set of synonyms corresponding to the keywords, then the corresponding keywords are extracted, and a text data keyword node is constructed and a lookup set is performed.

[0097] Optionally, the aforementioned grouping module 33 is specifically used for:

[0098] Initialize the text data keyword nodes and perform a lookup;

[0099] The first keyword node and the second keyword node are determined from the text data keyword node union set, wherein the first keyword node and the second keyword node are two text data keyword topology nodes connected by an edge;

[0100] Search for the root nodes of the first keyword node and the second keyword node, and merge the root nodes of the first keyword node and the second keyword node.

[0101] Traverse all node connections in the text data keyword node union set and determine the root node merging result for all first and second nodes.

[0102] Optionally, the grouping module 33 described above is specifically used for:

[0103] Traverse the connected components corresponding to the text data keyword nodes in the union set, and determine whether the current keyword is consistent with the connected components corresponding to the text data keyword nodes in the union set according to the association relationship between each node and the root node in the union set.

[0104] If the current keyword matches the connected component in the disjoint-set data structure of the text data keyword nodes, then the keyword nodes in the disjoint-set data structure of the text data nodes are grouped to obtain the grouping results.

[0105] Optionally, the grouping module 33 described above is also used for:

[0106] Based on the grouping results, determine the regional characteristics and weights of the keywords in the text data;

[0107] Based on the regional characteristics and weights of the keywords, the keyword categories of the text data are determined.

[0108] Optionally, the storage module 34 described above is specifically used for:

[0109] Based on the keyword categories of the text data, determine whether the keyword categories of the next text data belong to keyword categories that violate preset rules;

[0110] If the keyword category of the next text data belongs to a keyword category that violates the preset rules, then the text data of the next text will be blocked.

[0111] If the keyword category of the next text data does not belong to the keyword category that violates the preset rules, the obtained keywords will be categorized and stored in the database, and the type of the next text data will be directly determined.

[0112] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0113] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 4As shown, the computer device in this embodiment includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in any of the above-described text data classification method embodiments.

[0114] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0115] In one embodiment, a computer-readable storage medium is provided that, when the instructions in the computer-readable storage medium are executed by a processor in a computer device, enables the computer device to perform the steps of any embodiment of the text data classification method disclosed herein, which will not be repeated here. The computer-readable storage medium may be non-volatile or volatile.

[0116] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0117] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0118] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0119] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0120] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0121] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for classifying text data, characterized in that, include: Obtain the text dataset to be classified, and extract the set of synonyms corresponding to the keywords in the text dataset to be classified; Based on the pre-defined mapping relationship between text data and keywords, and the set of synonyms corresponding to the keywords, a text data keyword node union-find set is constructed; Traverse the connected components corresponding to the keyword nodes in the text data disjoint-setup, and group the keyword nodes in the text data keyword node disjoint-setup according to the association relationship between each node and the root node, so as to determine the keyword category of the text data based on the grouping results; Based on the keyword categories of the text data, the obtained keywords are categorized and stored in the database; Before traversing the text data keyword nodes and finding the corresponding connected components in the lookup set, the process includes: Initialize the text data keyword nodes and perform a lookup; The first keyword node and the second keyword node are determined from the text data keyword node union set, wherein the first keyword node and the second keyword node are two text data keyword topology nodes connected by an edge; Search the root nodes of the first keyword node and the second keyword node, and merge the root nodes of the first keyword node and the second keyword node; Traverse the keyword nodes of the text data and find all the node connections in the set, and determine the root node merging result of all first nodes and second nodes; The process of determining the keyword category of text data based on the grouping results includes: Based on the grouping results, determine the regional characteristics and weights of the keywords in the text data; Based on the regional characteristics and weights of the keywords, the keyword categories of the text data are determined; The construction of a text data keyword node disjoint-set data structure based on the pre-defined mapping relationship between text data and keywords, and the set of synonyms corresponding to the keywords, includes: Create a full content table of the text data, which includes keywords, time, identifiers, and storage location; Based on the pre-defined mapping relationship between text data and keywords, determine whether the keywords in the full content table of the text data are consistent with the set of synonyms corresponding to the keywords; If the keywords in the full content table of the text data are consistent with the set of synonyms corresponding to the keywords, then the corresponding keywords are extracted, and a text data keyword node is constructed and a lookup set is performed.

2. The text data classification method as described in claim 1, characterized in that, The step of extracting the set of synonyms corresponding to the keywords in the text dataset to be classified includes: Feature extraction is performed on the keywords to obtain the keyword features; Determine whether the similarity of keyword features between any two keywords is less than a preset similarity threshold; If the similarity of the keyword features of any two keywords is less than a preset similarity threshold, then these two keywords are identified as synonyms. This process continues until all keywords in the text data are identified, thereby determining the set of synonyms corresponding to the keywords in the categorized text dataset.

3. The text data classification method as described in claim 1, characterized in that, The process of traversing the connected components corresponding to the disjoint-setup of the text data keyword nodes, and grouping the keyword nodes in the disjoint-setup according to the association relationship between each node and the root node, includes: Traverse the connected components corresponding to the text data keyword nodes in the union set, and determine whether the current keyword is consistent with the connected components corresponding to the text data keyword nodes in the union set according to the association relationship between each node and the root node in the union set. If the current keyword matches the corresponding connected component in the text data keyword node union set, then the keyword nodes in the text data keyword node union set are grouped to obtain the grouping result.

4. The text data classification method as described in claim 1, characterized in that, The process of categorizing and storing keywords in the database based on the keyword categories of the text data includes: Based on the keyword categories of the text data, determine whether the keyword categories of the next text data belong to keyword categories that violate preset rules; If the keyword category of the next text data belongs to a keyword category that violates the preset rules, then the text data of the next text will be blocked. If the keyword category of the next text data does not belong to the keyword category that violates the preset rules, the obtained keywords will be categorized and stored in the database, and the type of the next text data will be directly determined.

5. A text data classification device, said text data classification device being based on the text data classification method according to any one of claims 1 to 4, characterized in that, include: The acquisition module is used to acquire the text dataset to be classified and extract the set of synonyms corresponding to the keywords in the text dataset to be classified. The construction module is used to construct text data keyword nodes and perform a lookup based on the pre-defined mapping relationship between text data and keywords and the set of synonyms corresponding to the keywords; The grouping module is used to traverse the connected components corresponding to the keyword nodes in the text data union set, and group the keyword nodes in the text data union set according to the association relationship between each node and the root node, so as to determine the keyword category of the text data based on the grouping results. The storage module is used to categorize and store the obtained keywords into the database based on the keyword categories of the text data.

6. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the text data classification method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the text data classification method as described in any one of claims 1 to 4.

Images