A text classification method and device

Abstract

The application discloses a text classification method and device, which is applied to the field of artificial intelligence technology and is used to improve the efficiency of text classification. The method comprises the following steps: splitting a text to be classified to obtain a plurality of subtexts; analyzing the subtexts in different formats by using different methods to determine the categories of the subtexts; and determining the category of the text to be classified according to the categories of the plurality of subtexts; wherein the analysis method corresponding to the subtexts in different formats can be preconfigured. Compared with the single method for text analysis and classification in the prior art, the scheme of the application adopts different analysis methods for subtexts in different formats, and the accuracy of text classification can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a text classification method and device. BACKGROUND

[0002] At present, the text classification method based on artificial intelligence is to extract the named entity in the text through a named entity recognition (NER) model, judge the category thereof, for example, judge whether the extracted named entity belongs to personnel, place name or amount, etc. Then, the text is classified based on the judged category, and the text after determining the category can be further associated or put into the corresponding set.

[0003] However, since the content of some text to be classified is relatively large, and the content at different positions in the text can be written in different formats. For example, the commodity description information in a letter of credit message includes a large amount of content, such as commodity name, commodity shipment details, price terms, customs number commodity composition, etc. Different content adopts different formats, such as the commodity name can be in the format of key-value pair, the price terms can be in the format of long text, and the commodity composition can be in the format of table or list. Therefore, when classifying this kind of text, if the simple NER model is still used to extract the named entity, the efficiency is relatively low, and a large amount of data needs to be used to train the NER model in advance, so that the NER model can recognize different formats of text, and the cost is relatively high. SUMMARY

[0004] In the exemplary embodiments of the present application, a text classification method and device are provided to improve the efficiency of text classification.

[0005] In a first aspect, the embodiments of the present application provide a text classification method, comprising:

[0006] Splitting the text to be classified to obtain at least two subtexts;

[0007] For any one subtext, determining the parsing mode corresponding to the format of the any one subtext according to the correspondence between the format and the parsing mode, and parsing the any one subtext based on the determined parsing mode to determine the category of the any one subtext;

[0008] Determining the category of the text to be classified based on the categories of the at least two subtexts.

[0009] Based on the above scheme, first, the text to be classified is split, and then different parsing methods are used to parse and determine the category of the different formats of the split subtext. Compared with the single named entity recognition method in the prior art, the efficiency is higher. Especially for the text including multiple contents and the content format is not single, the above method can greatly improve the speed and accuracy of text classification. And there is no need to train the recognition model in advance, which saves the cost of text classification.

[0010] In some embodiments, before determining the parsing method corresponding to the format of the arbitrary one subtext according to the correspondence between the preconfigured format and the parsing method, the method further comprises:

[0011] Converting the arbitrary one subtext into a subtext vector;

[0012] Inputting the subtext vector into a pre-trained recurrent neural network to obtain the probability of each format corresponding to the subtext vector;

[0013] Determining the format of the arbitrary one subtext according to the probability of each format corresponding to the subtext vector.

[0014] In some embodiments, the determining the format of the arbitrary one subtext according to the probability of each format corresponding to the subtext vector comprises:

[0015] Taking the format with the maximum probability in the probability of each format corresponding to the subtext vector as the format of the arbitrary one subtext; or,

[0016] Determining the format of the arbitrary one subtext according to the format of other subtexts in the at least one subtext except the arbitrary one subtext and the probability of each format corresponding to the subtext vector through the Viterbi algorithm.

[0017] In some embodiments, before converting the arbitrary one subtext into a subtext vector, the method further comprises:

[0018] Determining that the arbitrary one subtext includes a to-be-corrected word in a pre-stored correction vocabulary set; the correction vocabulary set includes the to-be-corrected word and a corrected word corresponding to the to-be-corrected word;

[0019] Correcting the to-be-corrected word in the arbitrary one subtext according to the correction vocabulary set.

[0020] In some embodiments, splitting the text to be classified comprises:

[0021] Splitting the text to be classified line by line.

[0022] In a second aspect, the embodiments of the present application provide a text classification device, comprising:

[0023] a splitting unit configured to split a text to be classified to obtain at least two subtexts;

[0024] a parsing unit configured to, for any one of the subtexts, determine, according to a correspondence between a format and a parsing manner, a parsing manner corresponding to the format of the any one of the subtexts, and parse the any one of the subtexts based on the determined parsing manner to determine a category of the any one of the subtexts;

[0025] The parsing unit is further configured to determine a category of the text to be classified based on the categories of the at least two subtexts.

[0026] In some embodiments, the parsing unit is further configured to:

[0027] convert the any one of the subtexts into a subtext vector;

[0028] input the subtext vector into a pre-trained recurrent neural network to obtain a probability of each format corresponding to the subtext vector;

[0029] determine the format of the any one of the subtexts according to the probability of each format corresponding to the subtext vector.

[0030] In some embodiments, the parsing unit is specifically configured to:

[0031] take a format with the largest probability in the probability of each format corresponding to the subtext vector as the format of the any one of the subtexts; or

[0032] determine the format of the any one of the subtexts according to the formats of other subtexts than the any one of the subtexts in the at least one subtext and the probability of each format corresponding to the subtext vector through a Viterbi algorithm.

[0033] In some embodiments, before converting the any one of the subtexts into a subtext vector, the parsing unit is further configured to:

[0034] determine, according to a pre-stored correction vocabulary set, that a to-be-corrected vocabulary included in the correction vocabulary set exists in the any one of the subtexts; the correction vocabulary set includes the to-be-corrected vocabulary and a corrected vocabulary corresponding to the to-be-corrected vocabulary;

[0035] correct the to-be-corrected vocabulary existing in the any one of the subtexts according to the correction vocabulary set.

[0036] In some embodiments, the splitting unit is specifically configured to:

[0037] The text to be classified is split line by line.

[0038] In a third aspect, an electronic device is provided. The electronic device includes a controller and a memory. The memory is configured to store computer-executable instructions, and the controller is configured to execute the computer-executable instructions in the memory to perform the operations of the method of any possible implementation of the first aspect using hardware resources in the controller.

[0039] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions which, when executed on a computer, cause the computer to perform the method of any of the aspects.

[0040] In a fifth aspect, a computer program product is provided. The computer program product includes computer program code, which, when executed on a computer, causes the computer to perform the method of any of the aspects.

[0041] In addition, the beneficial effects of the second aspect to the fifth aspect can refer to the beneficial effects as described in the first aspect, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application.

[0043] Figure 1 A text classification method flowchart provided by the embodiments of the present application;

[0044] Figure 2A A schematic diagram of a text to be classified before splitting provided by the embodiments of the present application;

[0045] Figure 2B A schematic diagram of a text to be classified after splitting provided by the embodiments of the present application;

[0046] Figure 3 A structural schematic diagram of a Word2Vec model provided by the embodiments of the present application;

[0047] Figure 4 A schematic diagram of a probability distribution of a plurality of word texts corresponding to a plurality of formats provided by the embodiments of the present application;

[0048] Figure 5 A structural schematic diagram of a text classification device provided by the embodiments of the present application;

[0049] Figure 6A structural schematic diagram of an electronic device is provided in the embodiments of the present application. DETAILED DESCRIPTION

[0050] The acquisition, storage, use, processing and the like of data in the technical solutions of the present application comply with relevant provisions of national laws and regulations.

[0051] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments described in the present application document, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the technical solutions of the present application.

[0052] The terms "first" and "second" in the specification and claims of the present application and the above drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprises" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but can optionally further comprise steps or units not listed, or can optionally further comprise other steps or units inherent to the process, method, product or device. "Multiple" in the present application can mean at least two, for example, can be two, three or more, and the embodiments of the present application are not limited.

[0053] In addition, the term "and / or" in this paper is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents an "or" relationship between the associated objects without special explanation.

[0054] In order to facilitate the understanding of the text classification method proposed in the embodiments of the present application, first, the technical terms involved in the present application are briefly introduced:

[0055] (1) Text classification: refers to automatically classifying and labeling text data or text set according to a certain classification system or standard. For example, according to a set of training texts that have been labeled, the relationship model between text features and text categories can be found, and then this learned relationship model is used to judge the categories of new texts. The current text classification method gradually changes from knowledge-based method to statistical and machine learning-based method.

[0056] (2) Information Extraction: This involves structuring the information contained in text and transforming it into a tabular organization. Specifically, the input to the information extraction system is the original text, and the output is information in a fixed format. Information is extracted from various documents and then integrated in a unified form. This is the main task of information extraction. The advantage of integrating information in a unified form is that it facilitates inspection and comparison. Information extraction technology does not attempt to fully understand the entire document, but only analyzes the parts of the document that contain relevant information. As for which information is relevant, that will depend on the domain scope defined during system design. Currently commonly used information extraction methods include extracting named entities from text based on Named Entity Recognition (NER) models. Named entities include phrases containing information such as names of people, places, organizations, dates, amounts, and times in the text.

[0057] (3) Named Entity Recognition (NER): Also known as proper name recognition, it refers to the identification of entities with specific meanings in text, mainly including names of people, places, organizations, proper nouns, etc. It usually includes two parts: entity boundary recognition; and determining the entity category (name of person, place, organization, or others).

[0058] (4) Natural Language Processing (NLP): NLP is a discipline that studies language problems in human-to-human communication and human-computer communication. Simply put, it involves computers accepting natural language input from users and processing and calculating it internally using human-defined algorithms to simulate human understanding of natural language and return the expected results to the user. The goal of NLP is to use computers to process large-scale natural language information instead of humans. It mainly includes automatic word segmentation, part-of-speech tagging, syntactic analysis, text classification, and information extraction. It is an interdisciplinary field of artificial intelligence, computer science, and information engineering, incorporating knowledge from statistics, linguistics, and other fields.

[0059] (5) Letter of Credit: This is a written document issued by a bank to an exporter (seller) at the request of the importer (buyer), guaranteeing payment for the goods. Under a letter of credit, the bank authorizes the exporter to draw a draft not exceeding the specified amount, with the bank or its designated bank as the drawee, and to attach the required shipping documents, and to collect the goods at the designated location on schedule, provided that the conditions stipulated in the letter of credit are met.

[0060] (6) 45A field of the credit message: the 45A field of the credit message is the commodity description information, and the content thereof includes commodity name, commodity shipping details, price terms, customs number, commodity composition, commodity color, packaging information, bulk identification, execution standard, freight, insurance premium, total quantity, total weight, total volume, certification terms statement, and referenced description, and the like. The commodity description information of the 45A field has the following characteristics: the text content is much, and the format types are much, including long text, key-value pair, table-like format, and list, and the like. In one position (for example, a certain row of the commodity description information of the 45A field), the format of the text is uniform (that is, there will not be multiple formats of content in one row of text, and generally, one row of text is of the same format, such as a row of text is of the key-value pair format).

[0061] (7) BIO tagging: each row of text is tagged as "B-X", "I-X", or "O". Among them, "B-X" indicates that this row of text belongs to X type, and as the beginning of the X type, this row of text in the whole text. "I-X" indicates that this row of text belongs to X type, and as the X type, this row of text is in the middle position of the whole text, and "O" indicates that it does not belong to any type.

[0062] (8) Word2Vec: a high-efficiency model for training word vectors. The core idea is that two words with similar contexts should also have similar word vectors. For example, banana and pear may often appear in the same context in a sentence, so the representation vectors of these two words should be more similar. The Word2Vec model can be regarded as a model that combines sentence context to correct word vectors, so that there is a correlation between the two context-related word vectors.

[0063] (9) Viterbi algorithm: also known as Viterbi analysis, it is a dynamic programming algorithm for finding the most likely sequence of hidden states (Viterbi path) that produces a sequence of observed events, especially in the context of Markov information sources and hidden Markov models. For example, in statistical syntax analysis, a dynamic programming algorithm can be used to find the most likely context-free derived string.

[0064] The text classification method in the related art is mostly based on the NER model. The NER model is used to extract the named entity in the text, and the category of each named entity is determined, that is, it is determined which category of the pre-configured category the extracted named entity belongs to, so as to determine the category of the text. However, the NER model is suitable for texts with less content. For texts with more content, especially for texts containing content in multiple formats, the efficiency of the NER model in named entity recognition is relatively low. In order to ensure that the NER model can identify all named entities in a text containing different formats, a large amount of training data is required to train the NER model to ensure the accuracy of the NER model, which is high in cost.

[0065] Therefore, the embodiments of the present application provide a text classification method and device. The text containing more content and format is split, the subtexts in different formats are split, then different analysis methods are used to determine the types of the subtexts obtained after splitting based on different formats, and finally the type of the text is determined according to the types of the multiple subtexts. The scheme proposed in the present application not only improves the efficiency of text analysis, but also improves the accuracy of text classification.

[0066] In order to facilitate the understanding of the scheme of the present application, see Figure 1 A flowchart of a text classification method is provided for the embodiments of the present application. It should be noted that the embodiments of the present application do not specifically limit the execution subject of the text classification method. For example, the text classification method can be executed by a computer, a mobile phone or other terminal, or can be executed by a server, a chip or other device. Figure 1 The method flow specifically includes:

[0067] 101, split the text to be classified to obtain at least two subtexts.

[0068] Among them, at least two subtexts belong to the text to be classified, and the formats of at least two subtexts can be the same or different. As an example, text A in at least two subtexts can be in the format of key-value pair, such as text A can be "commodity: bopet film" or "quantity: 23972.00 kgs (±10pct)". Text B in at least two subtexts, which is different from text A, can be a long text type, such as text B can be "more or less 5pct on quantity and amount for each item and in total acceptable".

[0069] 102, according to the pre-configured correspondence between the format and the analysis method, determine the corresponding analysis method for any one subtext.

[0070] Optionally, different formats can adopt different parsing manners. The correspondence between the format and the parsing manner can be preconfigured, and after the format of a certain subtext is determined, the parsing manner corresponding to the format of the subtext can be determined according to the preconfigured correspondence.

[0071] 103. parsing the arbitrary subtext based on the determined parsing manner to determine the category of the arbitrary subtext.

[0072] For ease of description, the first subtext in the at least two subtexts is taken as an example for introduction, and the first subtext is an arbitrary subtext in the at least two subtexts. After the format of the first subtext is determined, the parsing manner corresponding to the format of the first subtext can be determined according to the preconfigured correspondence between the format and the parsing manner, and then the first subtext is parsed according to the determined parsing manner to obtain the category of the first subtext.

[0073] For example, if the first subtext is a long text format subtext, a pre-established vocabulary library can be matched with the first subtext, the vocabulary library including named entities of various categories, and if a certain word or a certain field or a certain number of words in the first subtext matches a certain named entity in the vocabulary library, the category of the first subtext can be determined as the category of the named entity.

[0074] 104. determining the category of the text to be classified based on the categories of the at least two subtexts.

[0075] In one possible case, if the categories of the at least two subtexts obtained by splitting the text to be classified are all category A, such as amount, the category of the text to be classified is also category A. In another possible case, if the categories of the at least two subtexts include N categories (such as the first subtext is category A, the second subtext is category 2, the third subtext is category 3, and so on), the category of the text to be classified is the N categories. For example, the categories of the at least two subtexts include amount, total, and date, and the category of the text to be classified is amount, total, and date.

[0076] Based on the above scheme, first, the text to be classified is split to obtain multiple subtexts, and the formats of the multiple subtexts are judged. For example, it is judged whether the subtext belongs to the key-value pair format or the long text format, etc. Further, different parsing methods are used for subtexts of different formats to determine the categories of the multiple subtexts. Finally, the category of the entire text to be classified is determined according to the categories of the multiple subtexts. Compared with the single named entity recognition method in the prior art, the efficiency is higher. Especially for texts including multiple contents and the content format is not a single format, the above method can greatly improve the speed and accuracy of text classification. Moreover, there is no need to train the recognition model in advance, which saves the cost of text classification.

[0077] In some embodiments, before splitting the text to be classified, the amount of content included in the text to be classified can be first evaluated. If it is determined that the text to be classified includes a large amount of content, the text to be classified is split, otherwise it can not be split. For example, the number of bytes included in the text to be classified can be first counted. If the number of bytes included in the text to be classified is greater than a set threshold, it can be considered that the text to be classified includes a large amount of content and needs to be split.

[0078] As an optional way, when the text to be classified is split, a line-by-line splitting method can be used. That is, each line of the text to be classified is taken as a subtext. Based on this method, when the text to be classified is the text of the 45A field of the credit report (i.e. the text here is the commodity description information), or the text similar to this kind of text, since the feature of this kind of text is that each line (or each paragraph) adopts a format, therefore, in this scenario, the line-by-line splitting method can guarantee that the subtexts of different formats are split to the maximum extent. As an example, taking the text to be classified as the 45A field of the credit report as an example, the text to be classified before splitting can be seen from the text shown in Figure 2A The multiple subtexts obtained after line-by-line splitting can be seen from Figure 2B .

[0079] Optionally, after the text to be classified is split, the formats of the multiple subtexts obtained after splitting can be first determined, and the BIO tagging method is used to tag the categories of the subtexts (the BIO tagging can be seen from the technical terms introduced above). Next, a method for determining the formats of the multiple subtexts provided by the embodiments of the present application is introduced:

[0080] As an optional way, any one of the plurality of subtexts obtained after splitting can be split again to obtain a plurality of characters. Taking the first subtext in the plurality of subtexts as an example, for example, the first subtext is: “They are movie stars”. The first subtext can be split into: “They”, “are”, “movie”, “stars”. In some embodiments, due to the possibility of spelling errors, after splitting the first subtext, error correction can be performed first. For example, if the first subtext is: “They are movie st ar s”, it can be seen that in the first subtext, “stars” is misspelled as “st ar s”, so error correction needs to be performed on the plurality of words obtained after splitting the first subtext. As an optional error correction method, correction can be performed according to a pre-configured correction vocabulary set, which includes a word to be corrected (such as: “st ar s”) and a corrected word corresponding thereto (such as: “stars”). The specific error correction process can be: determining whether the plurality of words obtained by splitting the first subtext includes a word to be corrected in the correction vocabulary set, and if so, correcting according to the correction vocabulary set. As another optional error correction method, the frequency of occurrence of different letter combinations can be counted, and when the frequency of occurrence of a letter combination is determined to be 1, the letter combination is determined to be a word. Continuing the above example, in the first subtext, the frequency of occurrence of “ar” is 2, and the probability of occurrence of “stars” is 1, so “stars” is a word, thereby completing error correction.

[0081] Further, after error correction is completed, the plurality of characters obtained by splitting the first subtext can be randomly combined into word groups to obtain a plurality of results. For example, the first combination result can be: “They are”, “movie stars”, and the second combination result can be: “They are movie”, “stars”. Of course, there are other combination results here, which will not be described one by one, and only the above two combination results are taken as examples. Optionally, according to a pre-constructed word group library, the probability of each word group being composed can be determined, and the accuracy of each combination result can be determined. For example, it is determined that the probability of “movie stars” as a word group appearing is 85%, and the probability of “They are movie” as a word group appearing is 51%. Then it can be determined that the accuracy of the first combination result is higher than that of the second combination result. In this way, the combination result with the highest accuracy can be determined from a plurality of combination results. Especially when it comes to product names, the above method can well determine the combination result with the highest probability in the word group combination corresponding to a specific product name.

[0082] After determining the combination result with the highest probability corresponding to the first subtext, each word group in the combination result with the highest probability can be further represented as a word vector, which can be a two-dimensional vector for example. The dimension of the word vector is not limited in the present application. Alternatively, the word vectors represented by the word groups included in the combination result can be input into a Word2Vec model (see the technical terms introduced above) for correction to obtain a plurality of associated word vectors. As an example, the structure of the Word2Vec model can refer to Figure 3 , which includes an input layer, a hidden layer, and an output layer. Alternatively, the word vectors represented by the word groups included in the combination result can be input into the input layer, corrected through the hidden layer, and output from the output layer as the plurality of corrected word vectors associated with each other. Further, the plurality of corrected word vectors can be input into a pre-trained recurrent neural network model to obtain the format of the first subtext composed of the plurality of word vectors.

[0083] In one possible implementation, the recurrent neural network model can include an embeding layer for converting the input plurality of word vectors into a plurality of word vector sequences. Then, the recurrent neural network performs probability prediction on the plurality of word vector sequences to obtain the probability that the first subtext composed of the plurality of word vectors belongs to a certain format. For example, after inputting the plurality of word vectors corresponding to the first subtext into the recurrent neural network, the recurrent neural network can output that the probability of the first subtext being in the key-value pair format is 31%, the probability of the first subtext being in the long text format is 58%, and so on. Alternatively, the format with the highest probability can be taken as the format of the first subtext.

[0084] As another alternative, after the recurrent neural network predicts the probability that each of a plurality of subtexts including the first subtext belongs to a certain format, the format of each subtext can be determined by the Viterbi algorithm in combination with the format probabilities of the plurality of subtexts. For example, the probabilities that adjacent first, second, third, and fourth subtexts belong to various formats are as follows Figure 4 It can be seen that the first subtext has the highest probability of belonging to format A, the second subtext has the highest probability of belonging to format B, the third subtext has the highest probability of belonging to format A, and the fourth subtext has the highest probability of belonging to format C (the second highest probability of belonging to format B). Based on this, it can be deduced by the Viterbi algorithm that a text in format A is generally followed by a text in format B, and the probability that the fourth subtext belongs to format B is not low, so it is determined that the fourth subtext should belong to format B.

[0085] The above describes the process of determining the format of the plurality of subtexts after the text to be classified is split into the plurality of subtexts. Optionally, after the format of the plurality of subtexts is determined, the parsing manner corresponding to the format of the subtext can be determined according to the preconfigured correspondence between each format and parsing manner, the subtext is parsed according to the determined parsing manner, and the category of each subtext is determined. The following describes the process of parsing in combination with specific embodiments.

[0086] In some embodiments, if the subtext is in the key-value pair format, for example, the subtext is: "TOTAL QUANTITY: 23,972.00 KGS (+ / - 10PCT)", the "key" in the key-value pair can be extracted as the category of the subtext, for example, "TOTAL QUANTITY" is the category of the subtext. Further, the "value" can be obtained as the specific numerical value under the category.

[0087] In some other embodiments, if the format of the subtext is in the table or list format, the table header of the table can be obtained as the category of the subtext. For example, if the table header includes the fields of "date" or "amount", it can be determined that the category of the subtext is "date" and "amount", and then the data of other rows (or columns) except the table header can be specifically extracted as the specific numerical value under the category.

[0088] In some other embodiments, if the format of the subtext is in the long text format, a preconfigured vocabulary library including the named entities of each category can be matched with the subtext. If a word in the subtext matches a named entity in the vocabulary library, it can be determined that the category of the subtext is the category of the named entity.

[0089] It should be noted that there are many formats of texts, each of which corresponds to a different parsing manner. Here, only the above three are taken as examples for description.

[0090] Based on the same concept as the above method, see Figure 5 A text classification device 500 is provided for the embodiments of the present application. The device 500 can perform each step in the above method, and thus will not be described here again. The device 500 includes a splitting unit 501 and a parsing unit 502.

[0091] The splitting unit 501 is configured to split the text to be classified to obtain at least two subtexts.

[0092] The parsing unit 502 is configured to, for any one subtext, determine the parsing manner corresponding to the format of the any one subtext according to the preconfigured correspondence between the format and the parsing manner, and parse the any one subtext based on the determined parsing manner to determine the category of the any one subtext.

[0093] The parsing unit 502 is further configured to determine the category of the text to be classified based on the categories of the at least two subtexts.

[0094] In some embodiments, the parsing unit 502 is further configured to:

[0095] convert the arbitrary subtext into a subtext vector;

[0096] input the subtext vector into a pre-trained recurrent neural network to obtain a probability of each format corresponding to the subtext vector;

[0097] determine the format of the arbitrary subtext according to the probability of each format corresponding to the subtext vector.

[0098] In some embodiments, the parsing unit 502 is specifically configured to:

[0099] take the format with the maximum probability in the probability of each format corresponding to the subtext vector as the format of the arbitrary subtext; or

[0100] determine the format of the arbitrary subtext according to the format of the other subtexts except the arbitrary subtext in the at least one subtext and the probability of each format corresponding to the subtext vector by using the Viterbi algorithm.

[0101] In some embodiments, before converting the arbitrary subtext into a subtext vector, the parsing unit 502 is further configured to:

[0102] determine, according to a pre-stored correction vocabulary set, that the arbitrary subtext contains a word to be corrected included in the correction vocabulary set; the correction vocabulary set includes the word to be corrected and a corrected word corresponding to the word to be corrected;

[0103] correct the word to be corrected in the arbitrary subtext according to the correction vocabulary set.

[0104] In some embodiments, the splitting unit 501 is specifically configured to:

[0105] split the text to be classified line by line.

[0106] Figure 6 An electronic device 600 structure schematic diagram provided by an embodiment of the present application is shown. The electronic device 600 in the embodiment of the present application can further include a communication interface 603, for example, a network port, and the electronic device can transmit data through the communication interface 603.

[0107] In the embodiments of the present application, the memory 602 stores instructions executable by the at least one controller 601, and the at least one controller 601, by executing the instructions stored in the memory 602, can be configured to perform the steps in the above method, for example, the controller 601 can implement the functions of the splitting unit 501 and the parsing unit 502 in the above method. Figure 5

[0108] The controller 601 is the control center of the electronic device, can connect each part of the whole electronic device through various interfaces and lines, and can run or execute the instructions stored in the memory 602 and call the data stored in the memory 602. Optionally, the controller 601 can include one or more processing units, and the controller 601 can integrate an application controller and a modem controller, wherein the application controller mainly processes the operating system and the application program, and the modem controller mainly processes the wireless communication. It can be understood that the above-mentioned modem controller can also not be integrated into the controller 601. In some embodiments, the controller 601 and the memory 602 can be implemented on the same chip, and in some embodiments, they can also be implemented on separate chips respectively.

[0109] The controller 601 can be a general controller, such as a central controller (CPU), a digital signal controller, an application specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general controller can be a microcontroller or any conventional controller. The steps performed by the data statistical platform disclosed in the embodiments of the present application can be directly executed by the hardware controller, or executed by the combination of hardware and software modules in the controller.

[0110] ​The memory 602, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 602 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 602 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used to store program instructions and / or data.

[0111] By designing and programming the controller 601, for example, the code corresponding to the training method of the neural network model introduced in the foregoing embodiments can be fixed in the chip, so that the chip can execute the steps of the foregoing neural network model training method at runtime. How to design and program the controller 601 is a technology known to those skilled in the art, and will not be described here.

[0112] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0113] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0114] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0115] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0116] While the preferred embodiments of the application have been described, additional variations and modifications can be employed by those skilled in the art. Therefore, the appended claims intend to cover all such modifications and variations as fall within the true spirit and scope of the application.

[0117] It is apparent that a number of modifications and changes can be made to the present application without departing from the true spirit and scope of the application. Accordingly, all such modifications and changes are intended to be included within the scope of the application as set forth in the following claims and the equivalents thereof.

Claims

1. A method of text classification, characterized by, The method comprises the following steps: splitting the text to be classified to obtain at least two subtexts; the formats of the at least two subtexts are the same or different; the at least two subtexts are obtained by splitting different formats of subtexts from the text to be classified; for any one of the subtexts, a format corresponding analysis method of the any one of the subtexts is determined according to a pre-configured correspondence between formats and analysis methods, and the any one of the subtexts is analyzed based on the determined analysis method to determine the category of the any one of the subtexts; the format includes a key-value pair format, a long text format, and a table format; different formats use different analysis methods; the category of the text to be classified is determined based on the categories of the at least two subtexts; the category of the text to be classified is determined based on the categories of the at least two subtexts, which comprises: if it is determined that the categories of the at least two subtexts obtained by splitting the text to be classified are all category A, then the category of the text to be classified is category A; if it is determined that the categories of the at least two subtexts include N categories, then the category of the text to be classified is the N categories; before determining the format corresponding analysis method of the any one of the subtexts according to the pre-configured correspondence between formats and analysis methods, the method further comprises: transforming the any one of the subtexts into a subtext vector; inputting the subtext vector into a pre-trained recurrent neural network to obtain the probability of each format corresponding to the subtext vector; the recurrent neural network comprises an embeding layer for transforming a plurality of input word vectors into a plurality of word vector sequences; determining the format of the any one of the subtexts according to the probability of each format corresponding to the subtext vector.

2. The method of claim 1, wherein, determining the format of the any one of the subtexts according to the probability of each format corresponding to the subtext vector, which comprises: taking the format with the maximum probability in the probability of each format corresponding to the subtext vector as the format of the any one of the subtexts; or determining the format of the any one of the subtexts according to the formats of other subtexts in the at least one subtext except the any one of the subtexts and the probability of each format corresponding to the subtext vector by using a Viterbi algorithm.

3. The method of claim 1, wherein, before transforming the any one of the subtexts into a subtext vector, the method further comprises: determining that a to-be-corrected word included in a pre-stored correction word set exists in the any one of the subtexts according to the correction word set; the correction word set includes the to-be-corrected word and a corrected word corresponding to the to-be-corrected word; correcting the to-be-corrected word existing in the any one of the subtexts according to the correction word set.

4. The method according to any one of claims 1 to 2, wherein The splitting the text to be classified comprises: splitting the text to be classified line by line.

5. A text classification apparatus characterized by comprising: The method comprises the following steps: a splitting unit is configured to split the text to be classified to obtain at least two subtexts; the formats of the at least two subtexts are the same or different; the at least two subtexts are obtained by splitting different formats of subtexts from the text to be classified; The parsing unit is configured to, for any one subtext, determine a parsing manner corresponding to a format of the any one subtext according to a correspondence between the format and the parsing manner, and parse the any one subtext based on the determined parsing manner to determine a category of the any one subtext; the format includes a key-value pair format, a long text format, and a table format; different formats adopt different parsing manners; The parsing unit is further configured to determine the category of the text to be classified based on the categories of the at least two subtexts. The parsing unit is further configured to: convert the any one subtext into a subtext vector; input the subtext vector into a pre-trained recurrent neural network to obtain a probability of each format corresponding to the subtext vector; the recurrent neural network includes an embeding layer configured to convert a plurality of input word vectors into a plurality of word vector sequences; determine the format of the any one subtext according to the probability of each format corresponding to the subtext vector; The parsing unit is specifically configured to: if the categories of the at least two subtexts obtained by splitting the text to be classified are all category A, the category of the text to be classified is category A; if the categories of the at least two subtexts include N categories, the category of the text to be classified is the N categories.

6. The apparatus of claim 5, wherein, The parsing unit is specifically configured to: take a format with the maximum probability in the probabilities of each format corresponding to the subtext vector as the format of the any one subtext; or determine the format of the any one subtext according to the formats of other subtexts in the at least one subtext except the any one subtext and the probabilities of each format corresponding to the subtext vector by using a Viterbi algorithm.

7. The apparatus of claim 5 or 6, wherein, Before converting the any one subtext into a subtext vector, the parsing unit is further configured to: determine, according to a pre-stored correction vocabulary set, that a to-be-corrected vocabulary included in the correction vocabulary set exists in the any one subtext; the correction vocabulary set includes the to-be-corrected vocabulary and a corrected vocabulary corresponding to the to-be-corrected vocabulary; correct the to-be-corrected vocabulary existing in the any one subtext according to the correction vocabulary set.

8. The apparatus of any one of claims 5-6, wherein, The splitting unit is specifically configured to: split the text to be classified line by line.

9. An electronic device, comprising: The electronic device includes a controller and a memory, The memory is configured to store computer programs or instructions; The controller is configured to execute the computer programs or instructions in the memory, so that the method in any one of claims 1-4 is executed.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and the computer executable instructions, when invoked by a computer, cause the computer to execute the method in any one of claims 1-4.

11. A computer program product, characterised in that, The computer program product includes computer program code, and the computer program code, when run by a computer, causes the computer to execute the method in any one of claims 1-4.

Images