A text classification method and system based on Markov chain data enhancement

Abstract

This application proposes a text classification method based on Markov chain data enhancement, including the following steps: a) obtaining open source data to traverse the sample text, segmenting the positive sample text, obtaining Key and Value, and generating a Markov chain; b) obtaining similar and opposite words of Key and Value to increase the number of data samples; c) combining subwords in the Markov chain to generate sentences; d) performing data enhancement on discrete words; f) using the training set, validation set, and test set to perform deep learning training, obtain the best model, and use the best model to perform text classification. On the one hand, this method segmented the positive sample text to generate Key and Value, increased the number of samples, and then reorganized the sentences, and then performed data enhancement on discrete words, thereby improving the accuracy of classification and avoiding misjudgment. On the other hand, based on deep learning, it strengthens model training, makes the model's prediction ability more accurate, and makes open source analysis faster and more effective.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and mainly to a text classification method and system based on Markov chain data enhancement. Background Art

[0002] With the rapid development of the internet and the rise of the web, expressing one's opinions on various open-source social platforms, such as Facebook, Twitter, Tieba, and Weibo, has become a major pleasure in people's lives and work. How to effectively organize and categorize this data, as well as conduct qualitative content analysis, has long been a topic of intense research in various fields. Some have used deep learning to train text information to achieve text classification in specific regions, while others have employed Markov chains to classify sentiment within Weibo content. However, these methods all have limitations. They are all based on a single sample of existing chat content, resulting in insufficient sample size. They also fail to correlate previous and subsequent conversations about the same topic with the same person, and fail to combine previous and subsequent conversations within the same topic. This can lead to misclassification of sparse content or content that contradicts one's own meaning. Therefore, designing a text classification method that can enhance training on text data from various open-source websites, forums, and social networks, effectively reducing the error rate, has become an urgent issue. Summary of the Invention

[0003] In response to the above technical problems in the prior art, the inventors proposed a text classification method and system based on Markov chain data enhancement.

[0004] According to one aspect of the present invention, a text classification method based on Markov chain data enhancement is proposed, comprising the following steps:

[0005] S1: Obtain open source analysis data, organize it to obtain positive and negative samples, traverse the positive samples and analyze the relationship between the subwords in the text, and obtain a Markov chain based on the key and value analysis, where the key is the previous subword and the value is the next subword. The previous and next subwords are saved in dictionary objects;

[0006] S2: Get words of the same type and opposite type of the subwords of Key or Value to increase the number of data samples of the Markov chain;

[0007] S3: For each subword in the dictionary, randomly find the next subword in the Markov chain and reassemble it to generate a sentence;

[0008] S4: Discrete words are enhanced with Markov data and classified based on the preceding and following subwords of the chat context;

[0009] S5: Obtain an optimal model through deep learning training, and use the optimal model to perform text classification.

[0010] The present invention adopts data enhancement based on Markov chain. In the same text, a Markov chain is first formed according to the relationship between the upper and lower words. The next subword is randomly found for each subword in the Markov chain to recombine and generate a sentence. In the sequence of random variables, the probability of the next state is only determined by the current state and is independent of the previous state. The similar words and the opposite words of the previous subword Key and the subsequent subword Value are obtained, the number of data samples is increased, the discrete word data is enhanced, and the problem of misjudgment caused by sparse data and the opposite of what is meant is solved. On the basis of data enhancement based on Markov chain, deep learning is used to strengthen model training, so that the prediction ability of the model is more accurate, and the best model is obtained, thereby improving the accuracy of text classification.

[0011] In a further embodiment, sources for obtaining similar words include a similar word library, a commonly used synonym library on the Internet, and a similar word library of professional vocabulary.

[0012] The database of similar words is collected and organized in the process of text analysis. The sources of similar words include similar word databases, commonly used synonym databases on the Internet, and similar word databases of professional vocabulary. This makes the expansion of similar words of the front and back subwords in the key-value pairs have a broader data foundation support and stronger applicability. The expansion of similar words increases the number of samples.

[0013] In a further embodiment, obtaining antonyms specifically includes: on the one hand, adding "not" or "not" in front of all verbs; on the other hand, obtaining antonyms of the preceding and following subwords from an antonym library.

[0014] Among them, the anti-class word library is expanded through commonly used dictionary words, and the acquisition and preservation of anti-class words increases the number of samples.

[0015] In a specific embodiment, the specific implementation method of performing Markov data enhancement on discrete words and classifying the preceding and following subwords in the chat context is as follows:

[0016] The preceding and following subwords and the frequencies of the preceding and following subwords corresponding to the discrete words in the positive sample text are counted, so as to relatively completely expand the discrete words.

[0017] Through the Markov chain, we can obtain the preceding and following subwords corresponding to discrete words in a passage, count the frequencies of the preceding and following subwords, sort them from most to least, and judge the meaning of the discrete words by combining the several subwords ranked in the front column, so as to classify the discrete words.

[0018] In a specific embodiment, the optimal model is obtained through deep learning training, specifically:

[0019] S51: Training using the preprocessed model;

[0020] S52: Organize training data;

[0021] S53: Perform multiple rounds of training;

[0022] S54: Use the validation set to obtain the best model;

[0023] S55: Test the model using the test set.

[0024] The preprocessing model uses the Chinese version of the BERT pretrained model. BERT is essentially a stack of trained transformers and decoders. BERT pretrained models typically contain larger feedforward neural networks (768 and 1024 hidden layers, respectively) and more attention heads (12 and 16, respectively). Fine-tuning based on the BERT pretrained model improves word and sentence representation and accelerates the application of deep learning in natural language processing. Deep learning training is performed using training, validation, and test sets, and the model is continuously adjusted to ultimately achieve the optimal model with high text classification accuracy.

[0025] In a specific embodiment, the method for organizing training data is: combining the data of manually labeled training samples and the final label, then organizing the training samples, screening out non-empty data, filtering out invalid characters, and truncating the data within a set length range to obtain organized training data.

[0026] In a specific embodiment, the method of the multiple rounds of training is:

[0027] Input the training data into the pre-trained model, fine-tune the pre-trained model, output the loss value and results, and update the trainable parameters by backpropagation through judging the results and adjusting the loss value. Set 100-1000 rounds of training.

[0028] Each round of training is an iteration based on the parameter adjustment of the previous round. Through multiple rounds of training, the training parameters can be continuously updated according to the results and loss values, so that the pre-trained model can output more accurate results.

[0029] In a specific embodiment, the best model is obtained using the validation set. The specific method is: the data of the manually labeled validation sample is also combined with the data and labels, the validation data is converted into a feature vector, and the validation data is fed into the model generated by the training process for model evaluation. The accuracy of the validation set is calculated by combining the recall rate and the precision rate, and the best model trained so far is determined and saved.

[0030] The recall rate is the correct proportion of the predicted results, and the precision rate is the correct proportion of all the data. The combination of the recall rate and the precision rate makes the calculated accuracy of the validation set more representative.

[0031] In a specific embodiment, the model is tested through a test set in the following way: for the best model that has been obtained, it is put into the actual use environment for open source analysis, batches of unknown data are obtained, the unknown data are converted into feature vectors, the final result is verified through the model, and whether the requirements are met is determined by the recall rate and precision rate.

[0032] The recall and precision rates are used to test whether the model meets the requirements. If so, it is used directly. If not, the problem samples are processed, and the problem sample data is collected and converted into training data, and then the training is strengthened again. In this way, the optimal model is finally obtained.

[0033] In a second aspect, the present application proposes a computer system of an electronic device, comprising one or more components, which, when operating, implement the method described in the first aspect of the present application.

[0034] In a third aspect, the present application further proposes a computer-readable storage medium on which one or more computer programs are stored, characterized in that when the one or more computer programs are executed by a computer processor, the method described in the first aspect of the present application is implemented.

[0035] Fourthly, this application also proposes a text classification system based on Markov chain data enhancement, including:

[0036] Markov chain module: This module is used to segment text into subwords to obtain a Markov chain, increase the number of data samples, perform data augmentation on discrete words, and recombine subwords to generate sentences.

[0037] Deep learning training module: configured to perform deep learning training on data samples to obtain the best model for text classification;

[0038] Statistical output module: This module is used to count and output the text classification results of data samples obtained by the optimal model.

[0039] The present application discloses a text classification method based on Markov chain data enhancement. On the basis of word segmentation and sentence reorganization based on Markov chain, the preceding and following subwords in the Markov chain are expanded with opposite-category words and similar words, thereby increasing the number of data samples, performing data enhancement on discrete words, and avoiding classification errors caused by paradox. On the basis of Markov enhancement, deep learning training is performed using a training set, a validation set, and a test set to obtain an optimal model, and the optimal model is used to perform text classification. The text classification method of the present application can solve the problems of data sparsity and difficulty in text classification caused by paradox, and can enhance text data of various open source websites, forums, and communities to achieve more accurate classification results, faster analysis speed, better results, and greater controllability. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and, together with the description, serve to explain the principles of the present invention. Other embodiments and many of the expected advantages of the embodiments will be readily apparent as they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with respect to each other. Like reference numerals designate corresponding similar parts.

[0041] Figure 1 A flowchart of a text classification method for data enhancement based on Markov chains is shown in the present application;

[0042] Figure 2 A flowchart of a deep learning training method for text classification based on Markov chain data enhancement is shown in the present application;

[0043] Figure 3a A schematic structural diagram of a computer system 600 of an electronic device according to an embodiment of the present invention is shown;

[0044] Figure 3b A schematic structural diagram of a text classification system 700 for data enhancement based on Markov chains according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0046] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] Figure 1 The flowchart of the text classification method based on Markov chain data enhancement is shown in the present application. Figure 1 As shown, the method specifically includes the following steps:

[0048] S1: Obtain open source analysis data, organize it to obtain positive and negative samples, traverse the positive samples and analyze the relationship between subwords in the text, and obtain a Markov chain based on the key and value analysis, where the key is the previous subword and the value is the next subword. The previous and next subwords are saved in dictionary objects;

[0049] S2: Get words of the same type and opposite type of the subwords of Key or Value to increase the number of data samples of the Markov chain;

[0050] S3: For each subword in the dictionary, randomly find the next subword in the Markov chain and reassemble it to generate a sentence;

[0051] S4: Discrete words are enhanced with Markov data and classified based on the preceding and following subwords of the chat context;

[0052] S5: Obtain the best model through deep learning training and use the best model for text classification.

[0053] The present invention adopts data enhancement based on Markov chain. In the positive sample text, a Markov chain is first formed according to the relationship between the upper and lower words. In the Markov chain, the next subword is randomly found for each subword to recombine and generate a sentence. In the sequence of random variables, the probability of the next state is determined only by the current state and is independent of the previous state. By obtaining similar words and antonyms of the previous and next subwords, the number of data samples is increased, thereby solving the problem of sparse data. At the same time, the discrete word data is enhanced to avoid misjudgment caused by saying the opposite of what is meant.

[0054] Based on data enhancement based on Markov chains, deep learning is used to strengthen model training, making the model's prediction ability more accurate and obtaining the best model, thereby improving the accuracy of text classification.

[0055] The positive sample text is segmented into the preceding and following subwords Key and Value to generate a Markov chain. The following is a specific example:

[0056] Eg: contents = ['You won first place', 'Where are you going today', 'Tomorrow we're going fishing', 'Yesterday we realized a dream', 'What time will you come over to play']

[0057] The following Markov chain is obtained through Key-Value analysis:

[0058] {"you": ["get", "go", "what time"],

[0059] "get": ["already"],

[0060] "already": ["first place"],

[0061] ……

[0062] "what time": ["come over"],

[0063] "come over": ["play"],

[0064] }

[0065] Exemplarily, using the Markov chain, take out the Key or Value in the key-value pair, and obtain words of the same type as this sub-word, so as to increase the data sample size of the Markov chain, eg:

[0066] {"first place": ["champion", "second place", "third place"]

[0067] "where": "what place"

[0068] }

[0069] Exemplarily, using the Markov chain, take out the Key or Value in the key-value pair, and obtain words of the opposite type as this sub-word, eg:

[0070] {"go": ["not go"]

[0071] "achieve": "not achieve"

[0072] "come over": "not come over"

[0073] "get" : "lose"

[0074] }

[0075] In the Markov chain, take out a Key as the first sub-word, and then randomly select the next sub-word from its Value, and the generation of sentences can be achieved. For example, the recombined sentences are as follows:

[0076] "Yesterday we achieved the first place"

[0077] "Today you got the first place"

[0078] "Today what time will you come over to play"

[0079] "Today where shall we go"

[0080] 'You won the championship today'

[0081] 'Where are we going today?'

[0082] 'You lost the championship today'

[0083] 'We're not going anywhere today'

[0084] In a further embodiment, sources for obtaining similar words include a similar word library, a commonly used synonym library on the Internet, and a similar word library of professional vocabulary.

[0085] The database of similar words is collected and organized in the process of text analysis. The sources of similar words include similar word databases, commonly used synonym databases on the Internet, and similar word databases of professional vocabulary. This makes the expansion of similar words of the front and back subwords in the key-value pairs have a broader data foundation support and stronger applicability. The expansion of similar words increases the number of samples.

[0086] In a further embodiment, obtaining antonyms specifically includes: on the one hand, adding "not" or "not" in front of all verbs; on the other hand, obtaining antonyms of the preceding and following subwords from an antonym library.

[0087] Among them, the anti-class word library is expanded through commonly used dictionary words, and the acquisition and preservation of anti-class words increases the number of samples.

[0088] In a specific embodiment, the specific implementation method of performing Markov data enhancement on discrete words and classifying them in combination with the preceding and following subwords of the chat context is as follows:

[0089] The preceding and following subwords and the frequencies of the preceding and following subwords corresponding to the discrete words in the positive sample text are counted, so as to relatively completely expand the discrete words.

[0090] Through the Markov chain, we can obtain the preceding and following subwords corresponding to discrete words in a passage, count the frequencies of the preceding and following subwords, sort them from most to least, and judge the meaning of the discrete words by combining the several subwords ranked in the front column, so as to classify the discrete words.

[0091] For example, if someone in a group chat says: "You beat someone beautifully" and "You scolded them beautifully", then we can get the subwords before and after the discrete word "beautiful", "beat" and "scold" appearing. At this time, another person in the group chat simply says: "beautiful". At this time, we can use the subwords summarized above to relatively completely expand the current discrete word, and understand that the complete meaning of the discrete word "beautiful" should be "beat someone beautifully" or "scold them beautifully", which solves the problem of not being able to make judgments on discrete words alone, thereby realizing the data enhancement function.

[0092] Figure 2 The flowchart of the deep learning training of the text classification method based on Markov chain data enhancement in this application is shown as follows: Figure 2 As shown,

[0093] Obtaining the best model through deep learning training involves the following steps:

[0094] S51: Training using the preprocessed model;

[0095] S52: Organize training data;

[0096] S53: Perform multiple rounds of training

[0097] S54: Use the validation set to obtain the best model;

[0098] S55: Test the model using the test set.

[0099] The preprocessing model uses the Chinese version of the BERT pretrained model. BERT is essentially a stack of trained transformers and decoders. BERT pretrained models typically contain larger feedforward neural networks (768 and 1024 hidden layers, respectively) and more attention heads (12 and 16, respectively). Fine-tuning based on the BERT pretrained model improves word and sentence representation and accelerates the application of deep learning in natural language processing. Deep learning training is performed using training, validation, and test sets, and the model is continuously adjusted to ultimately achieve the optimal model with high text classification accuracy.

[0100] In a specific embodiment, the method for organizing training data is: combining the data of manually labeled training samples and the final label, then organizing the training samples, screening out non-empty data, filtering out invalid characters, and truncating the data within a set length range to obtain organized training data.

[0101] In a further embodiment, the specific method of organizing the training samples is:

[0102] The maximum length of a text is set to 128 characters. Insufficient data is padded with 0s, and the excess length is truncated. [CLS], sentence segmentation marker [SEP], and unknown character [UNK] are added to the beginning of the sentence. Then, a strategy is adopted to use [MASK] to mask some words in the sentence. The text data is further converted into feature vectors. Then, the training sample data is reorganized. 5% of each training sample is randomly replaced with incorrect entities, 15% masks the relationship between tokens and entities, and 80% remains unchanged. Finally, the training samples are packaged for training.

[0103] In a specific embodiment, the method for performing multiple rounds of training is specifically as follows:

[0104] Input the training data into the pre-trained model, fine-tune the pre-trained model, output the loss value and results, and update the trainable parameters by backpropagation through judging the results and adjusting the loss value. Set 100-1000 rounds of training.

[0105] Each round of training is an iteration based on the parameter adjustment of the previous round. Through multiple rounds of training, the training parameters can be continuously updated according to the results and loss values, so that the pre-trained model can output more accurate results.

[0106] In a specific embodiment, the best model is obtained using the validation set in the following manner:

[0107] The data and labels of the manually labeled verification samples are also combined. The verification data is converted into feature vectors and fed into the model generated by the training process for model evaluation. The accuracy of the verification set is calculated by combining the recall rate and the precision rate, and the best model trained so far is determined and saved.

[0108] The recall rate is the correct proportion of the predicted results, and the precision rate is the correct proportion of all the data. The combination of the recall rate and the precision rate makes the calculated accuracy of the validation set more representative.

[0109] In a specific embodiment, the model is tested through a test set in the following way: for the best model that has been obtained, it is put into the actual use environment for open source analysis, batches of unknown data are obtained, the unknown data are converted into feature vectors, the final result is verified through the model, and whether the requirements are met is determined by the recall rate and precision rate.

[0110] Test whether the requirements are met by using the recall rate and precision rate. If the requirements are met, it will be used directly; if not, the problem samples will be processed, the problem sample data will be centrally obtained, converted into training data, and strengthened training will be carried out again, so as to finally obtain the required optimal model.

[0111] Reference below Figure 3a , which shows a structural diagram of a computer system 600 suitable for implementing an electronic device of an embodiment of the present application. Figure 3a The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0112] like Figure 3aAs shown, computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of system 600 are also stored in RAM 603. CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0113] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, mouse, and the like; an output section 606 including a liquid crystal display (LCD) and speakers; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. Removable media 611, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, and the like, is installed in the drive 610 as needed, so that computer programs read from the removable media can be installed in the storage section 608 as needed.

[0114] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for executing the method illustrated in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 609 and / or installed from removable media 611. When executed by the central processing unit (CPU) 601, the computer program performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium described herein can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take various forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0115] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0116] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0117] The modules involved in the embodiments described in this application may be implemented in software or hardware. The units described may also be provided in a processor, and the names of these units do not, in certain circumstances, constitute limitations on the units themselves.

[0118] Embodiments of the present invention also relate to a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer processor, implements the method described above. The computer program includes program code for executing the method shown in the flowchart. It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable medium, or any combination thereof.

[0119] Figure 3bFIG. 7 shows a schematic diagram of a text classification system 700 for data enhancement based on a Markov chain according to an embodiment of the present application. Figure 3b As shown, it includes a Markov chain module 701, a deep learning training module 702 and a statistical output module 703.

[0120] Markov chain module 701: configured to segment the text into preceding and following subwords and obtain a Markov chain, increase the number of data samples, perform data enhancement on discrete words, and recombine the preceding and following subwords to generate sentences;

[0121] Deep learning training module 702: configured to perform deep learning training on data samples to obtain the best model for text classification;

[0122] Statistics output module 703: configured to count and output text classification results of data samples obtained through the best model.

[0123] The present application discloses a text classification method based on Markov chain data enhancement. On the basis of word segmentation and sentence reorganization based on Markov chain, the preceding and following subwords in the Markov chain are expanded with opposite-class words and similar words, thereby increasing the number of data samples, performing data enhancement on discrete words, avoiding classification errors caused by paradox, and performing deep learning training using training sets, validation sets, and test sets on the basis of Markov enhancement to obtain the optimal model. The text classification method of the present application can solve the problems of data sparsity and difficulty in text classification caused by paradox, and can enhance text data of various open source websites, forums, and communities to achieve more accurate classification results, faster analysis speed, better results, and greater controllability.

[0124] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention herein is not limited to technical solutions formed by a specific combination of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A text classification method based on Markov chain data enhancement, characterized in that: The method comprises the following steps: S1: Obtain open source analysis data, organize and obtain positive samples and negative samples, traverse the positive samples and analyze the relationship between subwords in the text, and obtain a Markov chain based on the key and value analysis, where the key is the front subword and the value is the back subword, and the front subword and the back subword are saved in a dictionary object; S2: Obtain words of the same type and opposite type of the subwords of the Key or Value to increase the number of data samples of the Markov chain; S3: For each subword in the dictionary, randomly find the next subword in the Markov chain and reassemble it to generate a sentence; S4: Markov data enhancement is performed on discrete words, and the preceding and following subwords of the chat context are combined for classification; The specific implementation method is: The preceding and following subwords and the frequencies of the preceding and following subwords corresponding to the discrete words in the positive sample text are counted, so as to relatively completely expand the discrete words; S5: Obtain an optimal model through deep learning training, and use the optimal model to perform text classification.

2. A text classification method based on Markov chain data enhancement according to claim 1, characterized in that: In S5, the optimal model is obtained through deep learning training as follows: S51: Training using the preprocessed model; S52: Organize training data; S53: Perform multiple rounds of training; S54: Use the validation set to obtain the best model; S55: Test the model using the test set.

3. A text classification method based on Markov chain data enhancement according to claim 2, characterized in that: In the step S52, the method of arranging the training data is as follows: The data of the manually labeled training samples and the final labels are combined, and then the training samples are sorted, non-empty data is screened out, invalid characters are filtered out, and the data is truncated within the set length range to obtain the sorted training data.

4. The text classification method based on Markov chain data enhancement according to claim 2, characterized in that: In S53, the method of performing multiple rounds of training is specifically as follows: Input the training data into the pre-trained model, fine-tune the pre-trained model, output the loss value and results, and update the trainable parameters by backpropagation through judging the results and adjusting the loss value. Set 100-1000 rounds of training.

5. The text classification method based on Markov chain data enhancement according to claim 2, characterized in that: In S54, the best model is obtained using the validation set, specifically in the following manner: The data and labels of the manually labeled verification samples are combined, and the verification data is converted into feature vectors. The verification data is fed into the model generated by the training process for model evaluation. The accuracy of the verification set is calculated by combining the recall rate and the precision rate, and the best model trained so far is determined and saved.

6. The text classification method based on Markov chain data enhancement according to claim 2, characterized in that: In S55, the model is verified by using the test set, specifically in the following manner: For the best model that has been obtained, it is put into the actual use environment for open source analysis, and batches of unknown data are obtained. The unknown data are converted into feature vectors, and the final results are verified through the model. The combination of recall rate and precision rate is used to determine whether it meets the requirements.

7. An electronic device, characterized in that: The device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the text classification method for data enhancement based on Markov chain according to any one of claims 1 to 6.

8. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that: When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 6 is implemented.

9. A text classification system based on Markov chain data enhancement, characterized in that: Applied to the method according to any one of claims 1 to 6, the system comprising: Markov chain module: This module is used to segment text into subwords to obtain a Markov chain, increase the number of data samples, perform data augmentation on discrete words, and recombine subwords to generate sentences. Deep learning training module: configured to perform deep learning training on data samples to obtain the best model for text classification; Statistical output module: This module is used to count and output the text classification results of data samples obtained by the optimal model.

Images