A deep learning based natural language processing system

Abstract

The application discloses a kind of natural language processing systems based on deep learning, including acquisition corpus unit, corpus preprocessing unit, feature engineering unit, feature selection unit, model training unit and model application unit, the acquisition corpus unit uses a text collection as corpus, the corpus source of the acquisition corpus unit has existing corpus, downloads corpus and scraping corpus, wherein the library source of scraping corpus is the whole network.The application has the advantages that the library source of scraping corpus is the whole network, the whole network is used as scraping corpus, and the WikiExtractor tool is used to divide the scraped corpus into files of specified size using the python command.The scraping time is linear, and DOM tree does not need to be built, and is independent of HTML tags.The scraped corpus is efficient and accurate, and the whole network corpus can be utilized, so that the language processing system can perform deep scraping corpus and model training and deep learning.Compared with the prior art, it has strong superiority.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of natural language processing, and particularly relates to a natural language processing system based on deep learning. BACKGROUND

[0002] Natural language processing is an important direction in the field of computer science and the field of artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers in natural language. Natural language processing is a science integrating linguistics, computer science and mathematics. Therefore, the research in this field will involve natural language, that is, the language used in daily life, so it is closely related to the research of linguistics, but there are important differences. Natural language processing is not generally studying natural language, but developing computer systems, especially software systems, that can effectively realize natural language communication. Thus, it is part of computer science.

[0003] In the prior art, when a natural language processing system acquires corpus, the corpus in a local corpus library and a downloaded corpus library is often acquired. The corpus vocabulary is limited, and the model training effect obtained during subsequent learning is limited, which can only be regarded as shallow learning. SUMMARY

[0004] (I) Technical problems solved

[0005] In view of the defects in the prior art, the application provides a natural language processing system based on deep learning, which has the advantages that the corpus library acquired can be divided into files of a specified size, the acquisition time is linear, DOM trees do not need to be built, and the acquisition of corpus is efficient and accurate, and the whole network corpus can be used, so that the natural language processing system can perform deep corpus acquisition, model training and deep learning, and has strong advantages compared with the prior art.

[0006] (II) Technical solutions

[0007] To achieve the above object, the application provides the following technical solutions: a natural language processing system based on deep learning, characterized by comprising an acquisition corpus unit, a corpus preprocessing unit, a feature engineering unit, a feature selection unit, a model training unit and a model application unit.

[0008] The acquisition corpus unit uses a text collection as a corpus library, and the corpus sources of the acquisition corpus unit include an existing corpus library, a downloaded corpus library and a crawled corpus library, wherein the source of the crawled corpus library is the whole network.

[0009] The WikiExtractor tool is applied when the corpus is crawled, the wikiextractor divides the corpus library into files of a specified size, and the format of the files is xml by default, and the specific format is as follows:

[0010] <docid="xxx" url="xxx" title="xxxx">

[0011] xxxxx

[0012]

[0013] Each generated file contains several entries, generating xml format corpus, using the following command:

[0014] WikiExtractor -o extracted_xml --process2 -b 512K --json zhwiki-latest-pages-articles.xml.bz2

[0015] -o is used to specify the output directory, --process is used to specify the number of processes used (default is 1), -b option is used to control the size of a single generated file (default is 1M, the larger the file, the more entries it contains), and the last parameter is the original compressed corpus file name to be processed; After the program runs, multiple subdirectories will be generated under the output directory, and some generated files will be generated under each subdirectory.

[0016] ["id":"xxx","url":"xxx","title":"xxx","text":"xxxxxx"]

[0017] text corresponds to the real content of the entry.

[0018] The technical scheme is adopted: the system, when learning, obtains the sources of the corpus unit from an existing corpus, a downloaded corpus and a crawled corpus, wherein the existing corpus is a local document accumulated, the downloaded corpus is a Baidu corpus and a Xinhua News Agency corpus, and the crawled corpus is a corpus source from the whole network. The corpus source is the whole network, and the WikiExtractor tool is used to divide the crawled corpus into files of a specified size using a python command. The crawling time is linear, and the DOM tree does not need to be built and is irrelevant to the HTML tag. The crawled corpus is efficient and accurate, and the whole network corpus can be used. The language processing system can perform deep crawling corpus and model training and deep learning. Compared with the prior art, the system has strong superiority.

[0019] Among them, the existing corpus is a local document accumulated, and the downloaded corpus is a Baidu corpus and a Xinhua News Agency corpus.

[0020] Preferably, the corpus preprocessing unit comprises a corpus cleaning module, a word segmentation module, a part-of-speech tagging module, and a stop word removal module, wherein the corpus cleaning module deletes noise data and performs data cleaning, manual deduplication, alignment, deletion, and tagging, or rule extraction content, regular expression matching, extraction according to part-of-speech and named entity, script writing, or code batch processing, and the word segmentation module divides text into words.

[0021] The above technical solutions are adopted: common word segmentation algorithms, string matching-based word segmentation methods, understanding-based word segmentation methods, statistical-based word segmentation methods, and rule-based word segmentation methods.

[0022] Preferably, the part-of-speech tagging module tags words with part-of-speech labels, and the stop word removal module removes words that do not contribute to the features of the text.

[0023] The above technical solutions are adopted: tagging words with part-of-speech labels such as adjectives, verbs, and nouns, which are required in tasks such as sentiment analysis and knowledge reasoning.

[0024] Preferably, the feature engineering unit represents word segmentation as a type that can be calculated by a computer, and the representation model includes bag-of-words model TF-IDF, word vector One-hot, and word2Vec.

[0025] Preferably, the feature selection unit selects appropriate and expressive features, and the feature selection methods include DF, MI, IG, CHI, WLLR, and WFO.

[0026] Preferably, the model training unit includes machine learning models and deep learning models, and the deep learning models include CNN, RNN, LSTM, Seq2Seq, and FastText.

[0027] The above technical solutions are adopted: pay attention to overfitting during model training, overfitting: performs very well on the training set but very poorly on the test set, and the solutions include increasing the training amount of data, adding regularization terms such as L1 regularization and L2 regularization, unreasonable feature selection, manual feature selection, and using a feature selection algorithm with the Dropout method.

[0028] The above technical solutions are adopted: pay attention to underfitting during model training, underfitting: the model cannot fit the data well, and the solutions include adding other feature terms, increasing the complexity of the model, such as adding more layers to the neural network, making the model more generalizable by adding a polynomial to the linear model, and reducing the regularization parameter, which is used to prevent overfitting, but now the model has underfitting, so the regularization parameter needs to be reduced.

[0029] Preferably, the model application unit is trained online, and the model is pickled and persisted after online training is completed.

[0030] (III) Beneficial Effects

[0031] Compared with the prior art, the natural language processing system based on deep learning has the following beneficial effects:

[0032] The natural language processing system based on deep learning acquires the sources of corpus units from existing corpus, downloaded corpus and crawled corpus when learning, wherein the existing corpus is accumulated local documents, the downloaded corpus is Baidu corpus and Xinhua News Agency corpus, and the crawled corpus is from the whole network. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A structure diagram of the natural language processing system based on deep learning is provided. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0035] Please refer to Figure 1 The natural language processing system based on deep learning comprises a corpus acquisition unit, a corpus preprocessing unit, a feature engineering unit, a feature selection unit, a model training unit and a model application unit.

[0036] The corpus acquisition unit uses a text collection as a corpus, and the sources of corpus of the corpus acquisition unit include existing corpus, downloaded corpus and crawled corpus, wherein the source of the crawled corpus is the whole network.

[0037] The WikiExtractor tool is applied when the corpus is crawled.

[0038] <docid="xxx"url="xxx"title="xxxx">

[0039] xxxxx

[0040] ;

[0041] Each generated file contains several entries, and the xml format corpus is generated using the following command:

[0042] WikiExtractor -o extracted_xml --process 2 -b 512K --json zhwiki-latest-pages-articles.xml.bz2

[0043] -o is used to specify the output directory, --process is used to specify the number of processes used (default is 1), -b option is used to control the size of the single generated file (default is 1M, the larger the file, the more entries it contains), and the last parameter is the name of the original compressed corpus file to be processed; After the program runs, multiple subdirectories will be generated under the output directory, and each subdirectory contains the generated files.

[0044] ["id":"xxx","url":"xxx","title":"xxx","text":"xxxxxx"]

[0045] text corresponds to the real content of the entry.

[0046] The system, when learning, obtains corpus units from existing corpus, downloaded corpus and crawled corpus, wherein the existing corpus is accumulated local document, the downloaded corpus is Baidu corpus and Xinhua News Agency corpus, and the focus is on the library source of the crawled corpus, which is the whole network. The whole network is used as the crawled corpus, and the WikiExtractor tool is used. The crawled corpus is divided into files of a specified size using the python command. The crawling time is linear, and the DOM tree does not need to be built and is irrelevant to the HTML tag. The crawled corpus is efficient and accurate. The whole network corpus can be used to make the language processing system perform deep crawling and model training and deep learning. Compared with the prior art, the system has strong superiority. The existing corpus is the accumulated local document, and the downloaded corpus is the Baidu corpus and the Xinhua News Agency corpus.

[0047] Embodiment 1: The corpus preprocessing unit includes a corpus cleaning module, a word segmentation module, a part-of-speech tagging module, and a stop word removal module. The corpus cleaning module deletes noise data and performs data cleaning, manual deduplication, alignment, deletion, and tagging, or rule extraction, regular expression matching, extraction based on part-of-speech and named entities, scripting, or code batch processing. The word segmentation module divides text into words. Common word segmentation algorithms include string matching-based word segmentation methods, understanding-based word segmentation methods, statistical-based word segmentation methods, and rule-based word segmentation methods.

[0048] Embodiment 2: The part-of-speech tagging module labels words with part-of-speech tags, and the stop word removal module removes words that do not contribute to the features of the text. Words are labeled with part-of-speech tags such as adjectives, verbs, and nouns, which are needed for tasks such as sentiment analysis and knowledge reasoning.

[0049] Embodiment 3: The feature engineering unit represents word segmentation as a type that can be calculated by a computer, and the representation model includes bag-of-words model TF-IDF, word vector One-hot, and word2Vec. The feature selection unit selects appropriate and expressive features, and the feature selection methods include DF, MI, IG, CHI, WLLR, and WFO. The model training unit includes machine learning models and deep learning models, and the deep learning models include CNN, RNN, LSTM, Seq2Seq, and FastText. Pay attention to overfitting during model training. Overfitting: performs well on the training set but performs poorly on the test set. Solutions include increasing the amount of data for training, adding regularization terms such as L1 regularization and L2 regularization, unreasonable feature selection, manual feature selection, and using the Dropout method.

[0050] Embodiment 4: Pay attention to the problem of underfitting during model training. Underfitting: the model cannot fit the data well. Solutions include adding other feature items, increasing the complexity of the model, such as adding more layers to the neural network, adding a polynomial to the linear model to make the model more general, and reducing the regularization parameter. The purpose of regularization is to prevent overfitting, but now the model has underfitting, so the regularization parameter needs to be reduced. The model application unit performs online training, and after online training is completed, the model is pickled and persisted.

[0051] In summary, the natural language processing system based on deep learning, when learning, the source of corpus unit is obtained from existing corpus, downloaded corpus and crawled corpus, wherein the existing corpus is accumulated local document, the downloaded corpus is baidu corpus and xinhua news agency corpus, and the focus is on the library source of the crawled corpus, which is the whole network, and the WikiExtractor tool is used to divide the crawled corpus into files of specified size using the python command, the crawling time is linear, and the DOM tree does not need to be built and is irrelevant to the HTML tag, the crawled corpus is efficient and accurate, the whole network corpus can be used, the language processing system can perform deep crawling corpus and model training and deep learning, compared with the prior art, the language processing system has strong superiority.

[0052] It should be noted that, in this document, the relationship terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including one" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0053] Although embodiments of the present application have been shown and described, it will be understood by those having ordinary skill in the art that various changes, modifications, alternatives and variations can be made thereto without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A deep learning based natural language processing system, characterized by, It comprises a corpus acquisition unit, a corpus preprocessing unit, a feature engineering unit, a feature selection unit, a model training unit and a model application unit. The corpus acquisition unit uses a text collection as a corpus, and the corpus sources of the corpus acquisition unit include existing corpus, downloaded corpus and crawled corpus, wherein the source of the crawled corpus is the whole network. When crawling the corpus, the WikiExtractor tool is applied. Wikiextractor divides the corpus into files of a specified size, and the default format of the files is xml format, and the specific format is as follows: <doc id="xxx" url="xxx" title="xxxx">< / doc> xxxxx ； Each generated file contains a number of entries, and the xml format corpus is generated by using the following command: WikiExtractor-o extracted_xml--process 2 -b 512K--json zhwiki-latest-pages-articles.xml.bz2 -o is used to specify the output directory, --process is used to specify the number of processes used, the default is 1, -b option is used to control the size of the single generated file, the default is 1M, the larger the file, the more entries it contains, and the last parameter is the original compressed corpus file name to be processed. After the program runs, multiple subdirectories will be generated in the output directory, and each subdirectory contains generated files. ["id": "xxx","url": "xxx","title": "xxx","text": "xxxxxx"] The text corresponds to the real content of the entry.

2. The natural language processing system based on deep learning according to claim 1, characterized in that: The corpus preprocessing unit comprises a corpus cleaning module, a word segmentation module, a part-of-speech tagging module and a stop word removal module, wherein the corpus cleaning module deletes noise data and performs data cleaning, manual deduplication, alignment, deletion and labeling, or rule extraction content, regular expression matching, according to the part-of-speech and named entity extraction, script or code batch processing, wherein the word segmentation module divides the text into words. 3.The natural language processing system based on deep learning of claim 2, characterized in that: The part-of-speech tagging module labels the words, and the stop word removal module removes words that do not contribute to the features of the text.

4. The natural language processing system based on deep learning according to claim 3, characterized in that: The feature engineering unit converts word segmentation into a type that can be calculated by a computer, and the representation model includes bag-of-words model TF-IDF, word vector One-hot and word2Vec.

5. The natural language processing system based on deep learning according to claim 4, characterized in that: The feature selection unit selects appropriate and expressive features, and the feature selection methods include DF, MI, IG, CHI, WLLR and WFO.

6. The natural language processing system based on deep learning according to claim 5, characterized in that: The model training unit comprises a machine learning model and a deep learning model, wherein the deep learning model comprises CNN, RNN, LSTM, Seq2Seq and FastText.

7. The natural language processing system based on deep learning according to claim 1, characterized in that: The model application unit performs online training, and the model is pickled after online training is completed.

Images