Text processing method, device, electronic device and computer-readable storage medium
Automatically filtering text data through the preset processor, the problem of text classification relying on stop word lists in the prior art is solved, and text classification with high accuracy and low resource consumption is achieved without field restrictions.
Patent Information
- Application Number
- CN202110815535.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-19
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2041-07-19
AI Technical Summary
The existing text classification methods rely on stop word lists and are easily restricted by the field to which text data belongs, resulting in low classification accuracy and huge computing resources consumption.
Use a preset processor to automatically classify, compare and calculate the text data, filter out appropriate text data from the text to be processed based on the target parameters, and directly remove the vocabulary that affects the text classification effect, avoid relying on stop word lists, and achieve unaffected Limitations of the field to which text data belongs.
It improves the accuracy and applicability of text classification, saves computing resources, reduces dependence on general stop word lists, and reduces the computational complexity of text classification.
Smart Images
Figure CN114610873B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology. Specifically, the present application relates to a text processing method, device, electronic device and computer-readable storage medium. Background Art
[0002] With the rapid development of the information age, the text data in the network is increasing. At present, in order to efficiently extract valuable text information from the network, text classification methods are often used to process these text data.
[0003] Text classification is one of the main research topics of Natural Language Processing (NLP). It is a process of automatically classifying and labeling text sets according to certain classification standards using computer programs. The text classification process mainly includes four aspects: text collection, text preprocessing, model training, and model application. Among them, text preprocessing plays a very important role in the text classification process. A good preprocessing result can make the subsequent model training converge quickly and achieve good results, save computing resources, and at the same time ensure the accuracy of the classification results, which is of great significance to production and application.
[0004] Generally speaking, existing text classification methods mainly include document segmentation, text segmentation, stop word removal, text feature extraction, word frequency statistics, text vectorization, model training and other steps. Among them, stop word removal is to remove some words in the document that are not actually helpful for classification. Usually, a more general stop word list is used as a reference, which is easily restricted by the field to which the text data belongs and the effect is relatively unstable; text feature extraction will take the current context into consideration, but the algorithm is complex and the amount of calculation is huge. It needs to be calculated for each text, which consumes a lot of computing resources and is time-consuming and labor-intensive; the result of word frequency statistics is heavily dependent on the effect of stop word removal and text feature extraction, which affects the accuracy of text classification results. Summary of the invention
[0005] The present application provides a text processing method, device, electronic device and computer-readable storage medium, which are used to solve the technical problem that the existing text processing method relies on a stop word list and is easily restricted by the field to which the text data belongs, resulting in low accuracy of text classification.
[0006] According to one aspect of the present application, a text processing method is provided, comprising:
[0007] Get the text to be processed;
[0008] Input the text to be processed into the preset processor to obtain the processing result;
[0009] Among them, the preset processor obtains the first target text in the text to be processed and classifies the first target text to obtain a processing result; the first target text includes the difference between the text to be processed and the second target text; the second target text includes text data whose target parameters in the text to be processed meet the preset parameter requirements.
[0010] Optionally, the method further comprises:
[0011] Get training text;
[0012] According to the training text, a preset processor is trained.
[0013] Optionally, according to the training text, a preset processor is trained, including:
[0014] Acquire a third target text in the training text; the third target text includes a difference set between the to-be-processed text and the fourth target text; the fourth target text includes text data whose target parameters in the to-be-processed text meet preset parameter requirements;
[0015] Convert the third target text into a text vector;
[0016] Input the text vector into the initial processor to obtain a first classification result;
[0017] The initial processor is reversely optimized according to the first classification result to obtain an optimized processor, until a preset processor that meets the preset accuracy requirement is obtained.
[0018] Optionally, obtaining a third target text in the training text includes:
[0019] determining a fourth target text in the training text;
[0020] The third target text is obtained by taking a difference set between the training text and the fourth target text.
[0021] Optionally, determining a fourth target text in the training text includes:
[0022] Perform word segmentation processing on the training text to obtain the first word segmentation text;
[0023] Obtain text category information of the first segmented text;
[0024] According to the text category information, determine the second segmented text whose target parameters meet the preset parameter requirements under each text category;
[0025] The intersection of the second segmented texts under each text category is taken to obtain the fourth target text.
[0026] Optionally, the target parameter comprises a frequency of occurrence.
[0027] According to another aspect of the present application, there is provided a text processing device, comprising:
[0028] A first acquisition module, used to acquire the text to be processed;
[0029] The processing module is used to input the text to be processed into a preset processor to obtain the processing result;
[0030] Among them, the preset processor obtains the first target text in the text to be processed and classifies the first target text to obtain a processing result; the first target text includes the difference between the text to be processed and the second target text; the second target text includes text data whose target parameters in the text to be processed meet the preset parameter requirements.
[0031] Optionally, the device further comprises:
[0032] The second acquisition module is used to acquire training text;
[0033] The training module is used to train and obtain a preset processor according to a training text.
[0034] Optionally, the training module includes:
[0035] A third acquisition module is used to acquire a third target text in the training text; the third target text includes a difference set between the text to be processed and the fourth target text; the fourth target text includes text data whose target parameters in the text to be processed meet the preset parameter requirements;
[0036] A conversion module, used for converting the third target text into a text vector;
[0037] A classification module, used for inputting the text vector into an initial processor to obtain a first classification result;
[0038] The optimization module is used to reversely optimize the initial processor according to the first classification result to obtain an optimized processor, until a preset processor that meets the preset accuracy requirement is obtained.
[0039] Optionally, the third acquisition module includes:
[0040] A determination module, used for determining a fourth target text in the training text;
[0041] The first calculation module is used to obtain a difference set between the training text and the fourth target text to obtain a third target text.
[0042] Optionally, the determination module includes:
[0043] A word segmentation module is used to perform word segmentation processing on the training text to obtain a first word segmentation text;
[0044] A fourth acquisition module, used to acquire text category information of the first segmented text;
[0045] A determination submodule is used to determine, according to the text category information, a second segmented text whose target parameters meet the preset parameter requirements under each text category;
[0046] The second calculation module is used to obtain the intersection of the second segmented texts under each text category to obtain a fourth target text.
[0047] Optionally, the target parameter comprises a frequency of occurrence.
[0048] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:
[0049] one or more processors;
[0050] Memory;
[0051] One or more applications, wherein the one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: execute the text processing method shown in the first aspect of the present application.
[0052] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the text processing method shown in the first aspect of the present application is implemented.
[0053] According to another aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementations of the first aspect above.
[0054] A text processing method provided by the present application is applied to obtain a text to be processed; the text to be processed is input into a preset processor to obtain a processing result; wherein the preset processor obtains a first target text in the text to be processed and classifies the first target text to obtain a processing result; the first target text includes a difference set between the text to be processed and a second target text; the second target text includes text data whose target parameters in the text to be processed meet the preset parameter requirements.
[0055] The present application automatically classifies, compares and calculates text data through a preset processor, selects a suitable first target text from the text to be processed based on the target parameters, and classifies the first target text to obtain a processing result. The text processing process does not rely on a stop word list, and directly removes words that affect the text classification effect based on the preset processor and the current text data, thereby achieving freedom from restrictions on the field to which the text data belongs, having strong applicability, and improving the accuracy of text classification while saving computing resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.
[0057] Figure 1 One of the flowcharts of a text processing method provided in an embodiment of the present application;
[0058] Figure 2 A second flowchart of a text processing method provided in an embodiment of the present application;
[0059] Figure 3 A structural diagram of a text processing device provided in an embodiment of the present application;
[0060] Figure 4 A schematic diagram of the structure of a text processing electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0061] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present invention.
[0062] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0063] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0064] In an existing application scenario, when performing text classification processing, a general stop word list is first used to remove stop words, and then text features are extracted through a large amount of calculations, and then the TF-IDF (term frequency-inverse document frequency) word frequency statistics method is used to perform word frequency statistics. Among them, when performing stop word removal processing, the general stop word list cannot be well applied to the field to which the current text data belongs, and it is impossible to filter out all the words that affect text classification. Text feature extraction requires calculation for each text, which consumes a lot of resources. Based on the poor results of stop word removal and text feature extraction, the word frequency statistics method cannot distinguish the differences between text data of different categories, which affects the effect of text classification.
[0065] The text processing method, device and electronic device provided in this application are intended to solve the above technical problems in the prior art.
[0066] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0067] The present application provides a text processing method, such as Figure 1 As shown, specifically including:
[0068] Step S101, obtaining the text to be processed.
[0069] The text to be processed includes text data after word segmentation. Before classifying the text data that the user needs to classify, a preprocessing operation is required, that is, the text data is firstly segmented to obtain the text to be processed.
[0070] The word segmentation processing can be done by using conventional word segmentation processing tools such as Jieba, SnowNLP, PkuSeg, THULAC, HanLP, etc., and there is no limitation here. When the type of text data belongs to an unconventional field, the user can customize the relevant vocabulary, and the word segmentation processing tool can perform word segmentation processing in combination with the customized vocabulary.
[0071] For example, the text to be processed by the user belongs to the field of biochemistry research and contains many rare proper nouns. Conventional word segmentation processing tools cannot distinguish proper nouns well. Users can customize proper nouns to help word segmentation processing tools process them, thereby improving the accuracy of word segmentation processing.
[0072] The word segmentation processing tool automatically divides the complete text paragraph into the text to be processed composed of words. For example, when the text data belongs to the user evaluation field, the text data input by the user is "The taste is really good, and my friends all like to eat it. I am still lucky today. There is no queue when I went there, and there are many people instantly after I sat down. But I feel that the hot pot dishes are a bit salty when I eat them, but the amount is definitely enough." Correspondingly, the text to be processed after being processed by the word segmentation processing tool is "The taste is really good, and my friends all like to eat it. I am still lucky today. There is no queue when I went there, and there are many people instantly after I sat down. But I feel that the hot pot dishes are a bit salty when I eat them, but the amount is definitely enough." It should be noted that the above text data content is only for example, and the text data in the embodiments of the present application is not limited to the user evaluation field, and the type of text data is not limited to any restrictions.
[0073] Step S102, inputting the text to be processed into a preset processor to obtain a processing result;
[0074] Among them, the preset processor obtains the first target text in the text to be processed and classifies the first target text to obtain a processing result; the first target text includes the difference between the text to be processed and the second target text; the second target text includes text data whose target parameters in the text to be processed meet the preset parameter requirements.
[0075] After obtaining the text to be processed through word segmentation, the text to be processed is input into a preset processor. The embodiment of the present application provides a preset processor for automatically classifying the text to be processed input by the user.
[0076] The preset processor performs preliminary classification processing on the text to be processed input by the user to obtain a first target text, wherein the first target text includes a difference set between the text to be processed and the second target text; and the second target text includes text data whose target parameters in the text to be processed meet the preset parameter requirements.
[0077] Specifically, the preset processor automatically determines the second target text based on the text to be processed, wherein the second target text includes text data whose target parameters meet the preset parameter requirements. Further, the preset processor determines the first target text based on the text to be processed and the second target text, wherein the first target text includes the difference set between the text to be processed and the second target text. The preset processor obtains the processing result by classifying the first target text.
[0078] The preset processor is pre-set with parameter requirements of the target parameter, which is used to determine the text data whose target parameter in the text to be processed meets the preset parameter requirements as the second target text. The target parameter can be the frequency of occurrence of the text data. The second target text automatically determined by the preset processor can be understood as text data that does not contribute to the text classification process or is of little significance. The preset processor needs to remove this type of text data to improve the accuracy of the text classification process.
[0079] After determining the second text data, the preset processor automatically calculates the difference between the text to be processed and the second text data as the first text data. In other words, the preset processor selects suitable text data from the text to be processed as the first text data based on the target parameter, and further obtains the processing result by classifying the first target text data.
[0080] Apply the text processing method provided in the embodiment of the present application to obtain the text to be processed; input the text to be processed into a preset processor to obtain a processing result; wherein the preset processor obtains a first target text in the text to be processed and classifies the first target text to obtain a processing result; the first target text includes a difference set between the text to be processed and the second target text; the second target text includes text data whose target parameters in the text to be processed meet the preset parameter requirements.
[0081] In an embodiment of the present application, text data is automatically classified, compared and calculated through a preset processor, a suitable first target text is screened out from the text to be processed based on the target parameters, and the first target text is classified and processed to obtain a processing result. The text processing process does not rely on a stop word list, and directly removes words that affect the text classification effect based on the preset processor and the current text data, thereby achieving freedom from restrictions on the field to which the text data belongs, having strong applicability, and improving the accuracy of text classification while saving computing resources.
[0082] Another text processing method is provided in the embodiment of the present application, such as Figure 2 As shown, specifically including:
[0083] Step S201, obtaining training text.
[0084] Obtain text data suitable for text classification training, that is, obtain training text.
[0085] Step S202: training a preset processor according to the training text.
[0086] A training text is obtained, and a preset processor is trained according to the training text. The preset processor performs preliminary classification processing on the training text to obtain a third target text. The third target text includes a difference set between the training text and the fourth target text; and the fourth target text includes text data in which target parameters in the training text meet the preset parameter requirements.
[0087] Step S203, obtaining the text to be processed.
[0088] The text to be processed includes text data after word segmentation. Before classifying the text data that the user needs to classify, a preprocessing operation is required, that is, the text data is firstly segmented to obtain the text to be processed.
[0089] The word segmentation processing can be done by using conventional word segmentation processing tools such as Jieba, SnowNLP, PkuSeg, THULAC, HanLP, etc., and there is no limitation here. When the type of text data belongs to an unconventional field, the user can customize the relevant vocabulary, and the word segmentation processing tool can perform word segmentation processing in combination with the customized vocabulary.
[0090] For example, the text to be processed by the user belongs to the field of biochemistry research and contains many rare proper nouns. Conventional word segmentation processing tools cannot distinguish proper nouns well. Users can customize proper nouns to help word segmentation processing tools process them, thereby improving the accuracy of word segmentation processing.
[0091] The word segmentation processing tool automatically divides the complete text paragraph into the text to be processed composed of words. For example, when the text data belongs to the user evaluation field, the text data input by the user is "The taste is really good, and my friends all like to eat it. I am still lucky today. There is no queue when I went there, and there are many people instantly after I sat down. But I feel that the hot pot dishes are a bit salty when I eat them, but the amount is definitely enough." Correspondingly, the text to be processed after being processed by the word segmentation processing tool is "The taste is really good, and my friends all like to eat it. I am still lucky today. There is no queue when I went there, and there are many people instantly after I sat down. But I feel that the hot pot dishes are a bit salty when I eat them, but the amount is definitely enough." It should be noted that the above text data content is only for example, and the text data in the embodiments of the present application is not limited to the user evaluation field, and the type of text data is not limited to any restrictions.
[0092] Step S204: input the text to be processed into a preset processor to obtain a processing result.
[0093] After obtaining the text to be processed through word segmentation, the text to be processed is input into a preset processor. The embodiment of the present application provides a preset processor for automatically classifying the text to be processed input by the user.
[0094] The preset processor performs preliminary classification processing on the text to be processed input by the user to obtain a first target text, wherein the first target text includes a difference set between the text to be processed and the second target text; and the second target text includes text data whose target parameters in the text to be processed meet the preset parameter requirements.
[0095] Specifically, the preset processor automatically determines the second target text based on the text to be processed, wherein the second target text includes text data whose target parameters meet the preset parameter requirements. Further, the preset processor determines the first target text based on the text to be processed and the second target text, wherein the first target text includes the difference set between the text to be processed and the second target text. The preset processor obtains the processing result by classifying the first target text.
[0096] The preset processor is pre-set with parameter requirements of the target parameter, which is used to determine the text data whose target parameter in the text to be processed meets the preset parameter requirements as the second target text. The target parameter can be the frequency of occurrence of the text data. The second target text automatically determined by the preset processor can be understood as text data that does not contribute to the text classification process or is of little significance. The preset processor needs to remove this type of text data to improve the accuracy of the text classification process.
[0097] After determining the second text data, the preset processor automatically calculates the difference between the text to be processed and the second text data as the first text data. In other words, the preset processor selects suitable text data from the text to be processed as the first text data based on the target parameter, and further obtains the processing result by classifying the first target text data.
[0098] Apply the text processing method provided in the embodiment of the present application to obtain training text; train a preset processor based on the training text; obtain a text to be processed; input the text to be processed into the preset processor to obtain a processing result.
[0099] The present application automatically classifies, compares and calculates text data through a preset processor, selects a suitable first target text from the text to be processed based on the target parameters, and classifies the first target text to obtain a processing result. The text processing process does not rely on a stop word list, and directly removes words that affect the text classification effect based on the preset processor and the current text data, thereby achieving freedom from restrictions on the field to which the text data belongs, having strong applicability, and improving the accuracy of text classification while saving computing resources.
[0100] In a preferred embodiment of the present application, a preset processor is trained according to a training text, including steps 1 to 4:
[0101] Step 1, obtaining a third target text in the training text; the third target text includes a difference set between the text to be processed and the fourth target text; the fourth target text includes text data whose target parameters in the text to be processed meet the preset parameter requirements;
[0102] A training text is obtained, and a preset processor is trained according to the training text. The preset processor performs preliminary classification processing on the training text to obtain a third target text. The third target text includes a difference set between the training text and the fourth target text; and the fourth target text includes text data in which target parameters in the training text meet the preset parameter requirements.
[0103] Specifically, the preset processor automatically determines a fourth target text based on the training text, wherein the fourth target text includes text data whose target parameters meet the preset parameter requirements. Further, the preset processor determines a third target text based on the training text and the fourth target text, wherein the third target text includes a difference set between the training text and the fourth target text.
[0104] The preset processor is pre-set with parameter requirements of the target parameter, which is used to determine the text data whose target parameter in the training text meets the preset parameter requirements as the fourth target text. The target parameter can be the frequency of occurrence of the text data. The fourth target text automatically determined by the preset processor can be understood as text data that does not contribute to the text classification process or is of little significance. The preset processor needs to remove this type of text data to improve the accuracy of the text classification process.
[0105] After determining the fourth text data, the preset processor automatically calculates the difference between the training text and the fourth text data as the third text data. In other words, the preset processor selects suitable text data from the training text as the third text data based on the target parameter.
[0106] Step 2, converting the third target text into a text vector;
[0107] The third target text is converted into a text vector. Specifically, the embodiment of the present application can input the third target text into the BERT pre-training model to implement a text vectorization operation, which is not limited here.
[0108] Text vectorization refers to representing text as a real number vector that can be recognized by a computer. According to the different granularity, the text feature representation can be divided into several levels: word, phrase, sentence or chapter. The methods of text vectorization are mainly divided into discrete representation and distributed representation.
[0109] Step 3, inputting the text vector into the initial processor to obtain a first classification result;
[0110] After the third target text is converted into a text vector through the text vectorization model, the text vector is input into the initial processor to obtain a first classification result.
[0111] Step 4: reversely optimize the initial processor according to the first classification result to obtain an optimized processor, until a preset processor that meets the preset accuracy requirement is obtained.
[0112] The initial processor is reversely optimized according to the first classification result to obtain an optimized processor, until a preset processor that meets the preset accuracy requirement is obtained. The initial processor can be built based on a neural network model.
[0113] In a preferred embodiment of the present application, obtaining a third target text in the training text includes:
[0114] A fourth target text in the training text is determined.
[0115] The third target text is obtained by taking a difference set between the training text and the fourth target text.
[0116] The fourth target text is determined, and the difference set is taken between the original training text and the fourth target text. In other words, the text data in the training text other than the text data included in the fourth target text is used as the third target text. Further, the third target text is converted into a text vector, and the text vector is input into the initial processor to obtain a first classification result; the initial processor is reversely optimized according to the first classification result to obtain an optimized processor, until a preset processor that meets the preset accuracy requirement is obtained.
[0117] In a preferred embodiment of the present application, determining the fourth target text in the training text includes:
[0118] Perform word segmentation on the training text to obtain the first word segmentation text.
[0119] Before training the preset processor according to the training text, it is necessary to perform a preprocessing operation on the training text, that is, first perform word segmentation processing on the training text data to obtain the first word segmentation text.
[0120] The word segmentation processing can be done by using conventional word segmentation processing tools such as Jieba, SnowNLP, PkuSeg, THULAC, HanLP, etc., and there is no limitation here. When the type of text data belongs to an unconventional field, the user can customize the relevant vocabulary, and the word segmentation processing tool can perform word segmentation processing in combination with the customized vocabulary.
[0121] For example, the training text entered by the user belongs to the field of biochemistry research and contains many rare proper nouns. Conventional word segmentation processing tools cannot distinguish proper nouns well. Users can customize proper nouns to help word segmentation processing tools process them, thereby improving the accuracy of word segmentation processing.
[0122] The word segmentation processing tool automatically divides the complete text paragraph into training texts composed of vocabulary. For example, when the text data belongs to the user evaluation field, the training text is "The taste is really good, and my friends all like to eat it. I was quite lucky today. There was no queue when I went there, and there were a lot of people instantly after I sat down. But I felt that the hot pot dishes were a bit salty when I ate them, but the amount was definitely enough." Correspondingly, the first word segmentation text after being processed by the word segmentation processing tool is "The taste is really good, and my friends all like to eat it. I was quite lucky today. There was no queue when I went there, and there were a lot of people instantly after I sat down. But I felt that the hot pot dishes were a bit salty when I ate them, but the amount was definitely enough." It should be noted that the above text data content is only for example, and the text data in the embodiments of the present application are not limited to the user evaluation field, and the type of text data is not limited to any restrictions.
[0123] Get the text category information of the first segmented text.
[0124] The text category information of the first segmented text is obtained. Specifically, the text category information can be obtained while obtaining the training text, or it can be manually defined based on the training text. The first segmented text is divided into different categories of text data according to the text category information of the first segmented text. For example, when the text data belongs to the field of user evaluation, at least two different categories can be divided according to the level of user evaluation, where category 1 includes "The taste is really good, and my friends like it. I was lucky today. There was no queue when I went there, and there were a lot of people after I sat down. But I felt that the hot pot was a bit salty when I ate it, but the amount was definitely enough." Category 2 includes "The amount given was too little, no one was full, and the soup was too salty, and the steak was a bit overcooked. In short, I was not very satisfied." At least two different text categories are determined, and the same type of text data is divided into the same category according to the text category information, and the text information under each text category is compared and calculated.
[0125] According to the text category information, the second segmented text whose target parameters meet the preset parameter requirements under each text category is determined.
[0126] According to the text category information, determine the second tokenized text in each text category where the target parameter meets the preset parameter requirements. Count the word frequencies of the text data in each text category. Specifically, the target parameter includes word frequency data, and the preset parameter requirements can be customized by the trainer. Count the preset number of high-frequency words in the text data of each text category. Here, high-frequency words refer to the preset number of words that appear among the top in the same text category. Determine the high-frequency words in each text category and classify them into the second tokenized text.
[0127] Take the intersection of the second tokenized text in each text category to obtain the fourth target text.
[0128] Compare the second tokenized text in different text categories, take the intersection of the second tokenized text of different categories, and calculate the occurrence frequencies of high-frequency words in different text categories. For example, the statistics show that the word "eat" appears 3,587 times in category 1 and belongs to the high-frequency words in category 1, and the word "eat" appears 16,432 times in category 2 and belongs to the high-frequency words in category 2. Since the word "eat" is a high-frequency word in both category 1 and category 2, it can be determined that this word is of little significance for text classification and classify this word into the fourth target text.
[0129] If the occurrence frequency of the counted high-frequency word in different text categories is higher than the preset threshold of the occurrence frequency, it means that this high-frequency word is of little significance for text classification processing and classify it into the fourth target text. Among them, the preset threshold of the occurrence frequency can be set by the trainer, and a reasonable threshold range can be 90% to 100%. For example, if the high-frequency word "eat" appears in the second tokenized text of 90% of different text categories, then classify this high-frequency word into the fourth target text. It should be noted that due to the large differences between text data in different fields, only an example is given for the threshold range here and there is no restriction. The fourth target text represents common words that are of little significance for text classification. During the process of training the preset processor, this type of text data needs to be discarded to improve the training effect.
[0130] Apply the text processing method provided in the embodiment of the present application to obtain a training text; perform word segmentation processing on the training text to obtain a first word segmentation text; obtain text category information of the first word segmentation text; determine the second word segmentation text whose target parameters meet the preset parameter requirements under each text category according to the text category information; take the intersection of the second word segmentation text under each text category to obtain a fourth target text. Take the difference set between the training text and the fourth target text to obtain a third target text. The third target text includes the difference set between the text to be processed and the fourth target text; the fourth target text includes text data whose target parameters meet the preset parameter requirements in the text to be processed; convert the third target text into a text vector; input the text vector into the initial processor to obtain a first classification result; reversely optimize the initial processor according to the first classification result to obtain an optimized processor, until a preset processor that meets the preset accuracy requirements is obtained.
[0131] This application combines the three steps of removing stop words, text feature extraction and word frequency statistics into one step, does not rely on the stop word list on the network, only based on existing text data, and is different from the traditional method of performing word frequency statistics in all texts, but classifies and statistics and then seeks intersection. By performing word segmentation and word frequency statistics on existing data, find out the words that do not contribute much to classification in the text data and remove them, greatly reducing the workload of subsequent text vectorization and model training, compared to using a general stop word list, it shows good adaptability for specific scenarios, can significantly improve the accuracy of text classification, and reduce the computing resources required for text classification. Not only does it reduce the dependence on the general stop word list, but it also eliminates the complex calculation of text feature extraction, avoids the errors that may be caused by traditional word frequency statistics, and improves the contrast between texts, which is conducive to distinguishing texts of different categories, while greatly reducing the length of each text, saving a lot of computing resources for subsequent text vectorization and model training.
[0132] The present application embodiment provides a text processing device, such as Figure 3 As shown, the device comprises:
[0133] A first acquisition module 301 is used to acquire a text to be processed;
[0134] The processing module 302 is used to input the text to be processed into a preset processor to obtain a processing result;
[0135] Among them, the preset processor obtains the first target text in the text to be processed and classifies the first target text to obtain a processing result; the first target text includes the difference between the text to be processed and the second target text; the second target text includes text data whose target parameters in the text to be processed meet the preset parameter requirements.
[0136] In one or more embodiments, the apparatus further comprises:
[0137] The second acquisition module is used to acquire training text;
[0138] The training module is used to train and obtain a preset processor according to a training text.
[0139] In one or more embodiments, the training module includes:
[0140] A third acquisition module is used to acquire a third target text in the training text; the third target text includes a difference set between the text to be processed and the fourth target text; the fourth target text includes text data whose target parameters in the text to be processed meet the preset parameter requirements;
[0141] A conversion module, used for converting the third target text into a text vector;
[0142] A classification module, used for inputting the text vector into an initial processor to obtain a first classification result;
[0143] The optimization module is used to reversely optimize the initial processor according to the first classification result to obtain an optimized processor, until a preset processor that meets the preset accuracy requirement is obtained.
[0144] In one or more embodiments, the third acquisition module includes:
[0145] A determination module, used for determining a fourth target text in the training text;
[0146] The first calculation module is used to obtain a difference set between the training text and the fourth target text to obtain a third target text.
[0147] In one or more embodiments, the determining module includes:
[0148] A word segmentation module is used to perform word segmentation processing on the training text to obtain a first word segmentation text;
[0149] A fourth acquisition module, used to acquire text category information of the first segmented text;
[0150] A determination submodule is used to determine, according to the text category information, a second segmented text whose target parameters meet the preset parameter requirements under each text category;
[0151] The second calculation module is used to obtain the intersection of the second segmented texts under each text category to obtain a fourth target text.
[0152] In one or more embodiments, the target parameter includes a frequency of occurrence.
[0153] The text processing device provided in the embodiment of the present application is used to obtain a text to be processed; the text to be processed is input into a preset processor to obtain a processing result; wherein the preset processor obtains a first target text in the text to be processed and classifies the first target text to obtain a processing result; the first target text includes a difference set between the text to be processed and the second target text; the second target text includes text data whose target parameters in the text to be processed meet the preset parameter requirements.
[0154] The present application automatically classifies, compares and calculates text data through a preset processor, selects a suitable first target text from the text to be processed based on the target parameters, and classifies the first target text to obtain a processing result. The text processing process does not rely on a stop word list, and directly removes words that affect the text classification effect based on the preset processor and the current text data, thereby achieving freedom from restrictions on the field to which the text data belongs, having strong applicability, and improving the accuracy of text classification while saving computing resources.
[0155] An electronic device is provided in an embodiment of the present application, and the electronic device includes: a memory and a processor; at least one program, which is stored in the memory and is used to be executed by the processor. Compared with the prior art, it can achieve: automatically classifying, comparing and calculating text data through a preset processor, screening out a suitable first target text from the text to be processed based on target parameters, classifying and processing the first target text to obtain a processing result, the text processing process does not rely on a stop word list, and directly removes words that affect the text classification effect based on the preset processor and the current text data, thereby achieving freedom from restrictions on the field to which the text data belongs, having strong applicability, and improving the accuracy of text classification while saving computing resources.
[0156] In an alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.
[0157] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0158] The bus 4002 may include a path for transmitting information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used, but it does not mean that there is only one bus or one type of bus.
[0159] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.
[0160] The memory 4003 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the contents shown in the above method embodiment.
[0161] The embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content of the aforementioned method embodiment. Compared with the prior art, the text data is automatically classified, compared and calculated by a preset processor, a suitable first target text is screened out from the text to be processed based on the target parameters, and the first target text is classified and processed to obtain a processing result. The text processing process does not rely on a stop word list, and directly removes words that affect the text classification effect based on the preset processor and the current text data, so that it is not limited by the field to which the text data belongs, has strong applicability, and improves the accuracy of text classification while saving computing resources.
[0162] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer device, enables the computer device to execute the coupon processing methods provided by the above-mentioned various method embodiments.
[0163] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0164] The above are only some embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A text processing method, characterized in that: include: Get training text; Acquire a third target text in the training text; The third target text includes a difference set between the training text and the fourth target text; the fourth target text includes text data in the training text whose appearance frequency is higher than a preset appearance frequency threshold, and the text data includes text data with practical significance; Converting the third target text into a text vector; Inputting the text vector into an initial processor to obtain a first classification result; Reversely optimizing the initial processor according to the first classification result to obtain an optimized processor, until the preset processor that meets the preset accuracy requirement is obtained; Get the text to be processed; Inputting the text to be processed into a preset processor to obtain a processing result; The preset processor obtains the first target text in the text to be processed and performs classification processing on the first target text to obtain the processing result; the first target text includes the difference set between the text to be processed and the second target text; the second target text includes the text data in the text to be processed whose occurrence frequency meets the preset parameter requirements; The preset processor processes the text to be processed through the following steps to obtain a processing result; Performing word segmentation processing on the text to be processed to obtain a first word segmentation text; Acquire text category information of the first segmented text, and divide the first segmented text into text data of different text categories according to the text category information; Determine, according to the text category information, a second segmented text whose target parameter meets the preset parameter requirements in the text data corresponding to each text category; Taking the intersection of the second word segmentation texts under each of the text categories to obtain the second target text; Calculate the difference between the to-be-processed text and the second target text to obtain the first target text; The first target text is classified and processed to obtain the processing result.
2. The text processing method according to claim 1, characterized in that: The obtaining of the third target text in the training text includes: Determining a fourth target text in the training text; A difference set is taken between the training text and the fourth target text to obtain the third target text.
3. The text processing method according to claim 2, characterized in that: The determining of the fourth target text in the training text includes: Performing word segmentation processing on the training text to obtain a first word segmentation text; Obtaining text category information of the first segmented text; Determine, according to the text category information, a second segmented text whose occurrence frequency meets the preset parameter requirements under each text category; The intersection of the second word segmentation texts under each of the text categories is taken to obtain the fourth target text.
4. A text processing device, characterized in that: include: The second acquisition module is used to acquire training text; A third acquisition module, used to acquire a third target text in the training text; The third target text includes a difference set between the training text and the fourth target text; the fourth target text includes text data in the training text whose appearance frequency is higher than a preset appearance frequency threshold, and the text data includes text data with practical significance; A conversion module, used for converting the third target text into a text vector; A classification module, used for inputting the text vector into an initial processor to obtain a first classification result; an optimization module, configured to reversely optimize the initial processor according to the first classification result to obtain an optimized processor, until the preset processor that meets the preset accuracy requirement is obtained; A first acquisition module, used to acquire the text to be processed; A processing module, used for inputting the text to be processed into a preset processor to obtain a processing result; The preset processor obtains the processing result by acquiring the first target text in the text to be processed and classifying the first target text; the first target text includes the difference between the text to be processed and the second target text; the second target text includes the text data whose target parameters in the text to be processed meet the preset parameter requirements; Wherein, the processing module is specifically used for: Performing word segmentation processing on the text to be processed to obtain a first word segmentation text; Acquire text category information of the first segmented text, and divide the first segmented text into text data of different text categories according to the text category information; Determine, according to the text category information, a second segmented text whose target parameter meets the preset parameter requirements in the text data corresponding to each text category; Taking the intersection of the second word segmentation texts under each of the text categories to obtain the second target text; Calculate the difference between the to-be-processed text and the second target text to obtain the first target text; The first target text is classified and processed to obtain the processing result.
5. An electronic device, characterized in that: The electronic device comprises: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs are configured to: execute the text processing method according to any one of claims 1-3.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text processing method according to any one of claims 1 to 3 is implemented.