A semantic fault-tolerant large-scale exploration and development BERT classification method

Abstract

The application provides a large-scale exploration and development BERT classification method based on semantic fault tolerance, which comprises the following steps: applying a certain proportion of random noise to literature and expanding the corpus; adopting a BERT algorithm to realize context-related first-order classification according to the expanded corpus, and obtaining classified sentences; and adopting an open-source Jieba word segmentation module to perform word segmentation on the classified sentences. Through input text, the number of sentences is expanded while the text chapter structure is kept unchanged, and the BERT algorithm is adopted to take a sentence pair as input, so that the memory of the 1-type grammar to the context before and after the sentence is realized, and the understanding of the chapter structure knowledge is indirectly realized.

Description

Technical Field

[0001] This invention relates to the field of exploration and development technology, and in particular to a BERT classification method for large-scale exploration and development based on semantic fault tolerance. Background Technology

[0002] To improve the accuracy of research result retrieval and achieve precise delivery for reports generated during the comprehensive research process of exploration and development, machine learning methods need to be adopted for unstructured reports to accurately classify the text for business purposes.

[0003] Traditional NLP business classification methods typically obtain labeled corpora by collecting a large number of samples and manually annotating them, and then build models using rule-based methods, statistical learning methods such as CRF, or deep learning methods such as LSTM.

[0004] Large-scale corpora are difficult to obtain for industry applications. Although public platforms such as CNKI and Baidu Wenku readily provide vast amounts of data, the data available for specific professional applications is very limited, such as literature on the comprehensive application of exploration and development in upstream oil operations. A search for "comprehensive application of exploration and development" on CNKI yields only 73 articles, including journal articles and dissertations; research documents from the Shengli Oilfield Geophysical Exploration Institute, including images, tables, and text files of various formats, only amount to 157. The difficulty in obtaining large-scale corpora for industry applications hinders effective large-scale classification technology research.

[0005] Traditional classification methods employ zero-order grammar, which is context-free grammar. This approach breaks down a document, regardless of length, into sentences, removing the relationships between them and assigning each sentence to the category of the entire document before classification. The most crucial knowledge lies in the relationships between sentences; therefore, zero-order grammar extracts the most essential knowledge framework of the entire document, discarding the most important high-level knowledge. Consequently, the processing methods employed by traditional classification methods cannot effectively express the core content of the entire document. Summary of the Invention

[0006] In view of the above problems, the present invention is proposed to provide a semantically fault-tolerant BERT classification method for large-scale exploration and development that overcomes or at least partially solves the above problems.

[0007] This invention provides a BERT classification method for large-scale exploration and development based on semantic fault tolerance, comprising:

[0008] Add a certain proportion of random noise to the literature and expand the corpus;

[0009] Based on the extended corpus, the BERT algorithm is used to perform context-dependent first-order classification to obtain classified sentences;

[0010] The categorized sentences were segmented using the open-source Jieba segmentation module.

[0011] Optionally, adding a certain proportion of random noise to the documents and expanding the corpus includes:

[0012] Random noise is added to the document to cause a certain percentage of word and phrase errors.

[0013] By reversing the search term correction technique, a new document is constructed, and the process is repeated to build a large amount of annotated corpus.

[0014] Optionally, the step of using the BERT algorithm to perform context-dependent first-order classification based on the extended corpus to obtain the classified sentences specifically includes:

[0015] The documents are input sequentially and categorized. The BERT algorithm is then used to perform first-order syntactic classification of the documents based on context and function.

[0016] BERT's paragraph reordering task involves shuffling the paragraphs of an article and then reordering them to restore the original text, enabling the algorithm to have a full and accurate understanding of the entire article.

[0017] Optionally, the step of segmenting the categorized statement using the open-source Jieba segmentation module specifically includes:

[0018] The sentence is segmented into words using the open-source Jieba word segmentation module, resulting in sentences in word units.

[0019] During the word segmentation process, a user dictionary should be added for specialized terms to improve the accuracy of word segmentation.

[0020] This invention provides a BERT classification method for large-scale exploration and development based on semantic error tolerance, comprising: adding a certain proportion of random noise to the documents and expanding the corpus; using the BERT algorithm to achieve context-dependent first-order classification based on the expanded corpus to obtain classified sentences; and using the open-source Jieba word segmentation module to segment the classified sentences. By adding errors to sentences in the input text while maintaining the text discourse structure, the amount of corpus is expanded. At the same time, the BERT algorithm is used with sentence pairs as input, realizing the memorization of the context of type 1 grammar, and indirectly realizing the understanding of discourse structure knowledge.

[0021] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a BERT classification method for large-scale exploration development based on semantic fault tolerance, according to the present invention.

[0024] Figure 2 This is a roadmap for the development of a BERT classification method for large-scale exploration based on semantic fault tolerance, as described in this invention.

[0025] Figure 3 This is a schematic diagram of corpus storage provided by the present invention;

[0026] Figure 4 A schematic diagram of the homonym dictionary provided by this invention;

[0027] Figure 5 A diagram illustrating the sentence-by-sentence arrangement of the documents provided for this invention;

[0028] Figure 6 This is a schematic diagram of a user-defined dictionary provided by the present invention;

[0029] Figure 7 This is a schematic diagram of the word segmentation results provided by the present invention;

[0030] Figure 8 This invention provides a schematic diagram for selecting incorrect words using pinyin.

[0031] Figure 9 This invention provides text with homophones incorrectly added;

[0032] Figure 10 The text with random errors provided by this invention;

[0033] Figure 11 This is a schematic diagram illustrating the principle of the BERT algorithm provided by the present invention. Detailed Implementation

[0034] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0035] The terms "comprising" and "having," and any variations thereof, in the specification, embodiments, claims, and drawings of this invention are intended to cover non-exclusive inclusion, such as including a series of steps or units.

[0036] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0037] like Figure 1 As shown, a large-scale exploration and development BERT classification method based on semantic fault tolerance includes:

[0038] Step 100: Add a certain proportion of random noise to the documents and expand the corpus;

[0039] Step 200: Use the BERT algorithm to perform context-dependent first-order classification based on the extended corpus to obtain the classified sentences;

[0040] Step 300: The classified statement is segmented using the open-source Jieba segmentation module.

[0041] First, noise is added to expand the corpus. A certain proportion of random noise is added to the documents to make a certain proportion of word and phrase errors, but without affecting human reading comprehension. Then, search term error correction technology is used to construct a new document and repeatedly build a large amount of annotated corpus.

[0042] For example, there are typos in the title "Tectonic Evolution Characteristics of Buried Hills in the Jizhong Depression and Their Petroleum Geological Significance" and "Tectonic Evolution Characteristics of Buried Hills in the Jizhong Depression and Their Petroleum Geological Significance". Although the typos may affect people's reading experience, they do not affect people's understanding of the core content of this document.

[0043] Secondly, the BERT algorithm is used to achieve context-dependent first-order classification. Documents are input sequentially and classified accordingly. The BERT algorithm's context-dependent capabilities are then utilized to achieve first-order syntactic classification of the documents. BERT's "paragraph reordering" task involves shuffling the paragraphs of an article and then reordering them to reconstruct the original text, achieving a comprehensive and accurate understanding of the entire text.

[0044] Reference Figure 2 The route for developing a large-scale BERT classification method based on semantic fault tolerance includes two layers: the first layer is the data layer, which provides the algorithm with a dictionary and corresponding corpus; the second layer is the data processing layer, which expands the input text and builds the BERT model.

[0045] Layer 1 consists of the 1-1 Original-Extended Corpus and the 1-2 Homophone Dictionary. The 1-1 Original-Extended Corpus is a subdirectory named after its class name. This subdirectory contains the original files, but not the extended files.Figure 3 as shown

[0046] 1-2 Homophone dictionary is a pair of homophone dictionary text files, including an open-source homophone dictionary and homophones that appear in the use of business personnel collected from the search terms of the internal knowledge management platform. For example Figure 4 as shown, among which the two records of talimu / Talimu and talimu / Talim are homophones obtained through search in the internal knowledge management platform

[0047] Layer 2 is the data processing layer, including reading literature 2-1, homophone misspelled words 2-2, random misspelled words 2-3, BERT model 2-4, and result evaluation 2-5

[0048] Reading literature 2-1 includes text reading, text type conversion, and word segmentation

[0049] In text reading, according to different text types, different reading methods and text processing methods are called as shown in Table 1. For the transformed text, it is first sorted in the order of paragraphs, and then for multiple sentences in a paragraph, the literature is segmented into sentences with the period "。" as the separator. Finally, the text of the entire literature is arranged in a column in the order of sequence. For example Figure 5 as shown

[0050] Table 1 File types and processing methods

[0051]

[0052] Finally, the open-source Jieba word segmentation module is used to segment the sentences to obtain sentences in units of words. In the word segmentation process, a user dictionary needs to be added for professional vocabulary to improve the accuracy of professional word segmentation. The method is to add a user-defined dictionary user.dic, and then call the user-defined dictionary through the following 2 sentences. User-defined dictionaries generally carry semantics. In addition to professional vocabulary, many are named entity objects, such as Well Bin 443 and Danjiasi Oilfield. If they are further broken down into smaller granularities, such as Danjiasi Oilfield being broken down into Danjiasi / Oilfield, although there is no problem from the perspective of word segmentation, it is difficult to understand from a business perspective. Therefore, the user dictionary should be sorted from the smallest application granularity

[0053] The result after word segmentation is as Figure 7 as shown

[0054] Homophone misspelled words 2-2 refer to the situation where when inputting Chinese in the Chinese Pinyin input method, the wrong word is often selected. For example, "exposure" is a commonly used word in the petroleum field but not in daily life. When inputting chulu, the following intelligent association selection options will appear, and wrong words will occur due to misselection

[0055] Error addition processing is performed at a probability of 5% of the total number of words in the whole text. For exampleFigure 8 As shown, for example, the total number of words is 119, and the number of words with homophone addition errors is 119 * 0.05 = 6. The words with homophone addition errors are as follows: south slope / difficult to break; outcrop / way out; towards the south / southern Hunan; oil source / having a source; deposition / achievement; two sets / two peaches. The text after homophone addition errors is as Figure 9 shown.

[0056] Random misspelling 2 - 3 means randomly changing the words in the whole text according to a ratio, mainly describing the situation of typing errors. For example, when typing chulu in a pinyin input method, since u and i are close, sometimes it may be typed wrongly as chuli. This kind of error is simulated with a probability of 1 / 1000. In the example, out of a total of 1290 pinyin letters, 1 is changed, changing si to su, and changing "sand four down" to "sand su down". The text after random addition of errors is as Figure 10 shown.

[0057] After random misspelling 2 - 3, the addition of errors to the text is completed, and finally a new document with added errors is obtained as the input for the BERT algorithm.

[0058] From the whole process, it can be seen that the errors at the word level do not change the overall structure of the document and the connotation of the sentences, so it does not affect the overall understanding of the document content by professionals. Although the ratio of 5% homophone addition errors needs to be adjusted.

[0059] BERT2 - 4 means using the BERT algorithm to model the input text and using it to predict unknown documents.

[0060] The full name of BERT is Bidirectional Encoder Representation from Transformers, that is, the Encoder of bidirectional Transformer, because a pure Decoder cannot obtain the information to be predicted. The main innovation points of the model are in the pre - train method, that is, using two methods, Masked LM and Next Sentence Prediction, to capture the representations at the word and sentence levels respectively.

[0061] The input of BERT is a single sentence or a pair of sentences. The actual input value includes three parts, namely token embedding word vectors, segment embedding sentence vectors, where each sentence has an overall embedding term for the sentence corresponding to each word, and position embedding position vectors. The sum of these three parts forms the final bert input vector, as Figure 11 shown.

[0062] Among them: Token Embeddings are word vectors, the first word is the CLS tag, which can be used for subsequent classification tasks.

[0063] Segment embeddings are used to distinguish between the two types of sentences because pre-training involves not only learning algorithms but also classification tasks with two sentences as input. Position embeddings are learned.

[0064] Obtain the open-source project package from https: / / github.com / applio / python-bert, then assemble the input data into sentence pairs, and perform computation to obtain the BERT model.

[0065] Results evaluation 2-5 refers to the validation of the algorithm through test results, using accuracy and recall for evaluation. Expanding the results across the three categories of literature, the test results are shown in Table 2. The average accuracy was 91.1%, which meets the engineering requirement of a primary accuracy of >85% for classification.

[0066] Table 2 Test Results

[0067]

[0068] Beneficial effects: This invention adds errors to sentences while keeping the text structure unchanged to expand the corpus. At the same time, it uses the BERT algorithm with sentence pairs as input to realize the memorization of context in Type 1 grammar, which indirectly realizes the understanding of text structure knowledge and achieves a classification accuracy of 91.1%, which meets the requirements of business classification in engineering.

[0069] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A BERT classification method for large-scale exploration and development based on semantic fault tolerance, characterized in that, The classification method includes: Add a certain proportion of random noise to the literature and expand the corpus, including: Random noise is added to the document to cause a certain percentage of word and phrase errors. By reversing the search term correction technology, a new document is constructed, and the process is repeated to build a large amount of annotated corpus. Based on the extended corpus, the BERT algorithm was used to perform context-dependent first-order classification, obtaining classified sentences, including: The documents are input sequentially and categorized. The BERT algorithm is then used to perform first-order syntactic classification of the documents based on contextual associations and functions. BERT's paragraph reordering task involves shuffling the paragraphs of an article and then reordering them to restore the original text, enabling the algorithm to have a full and accurate understanding of the entire article. The categorized sentences were segmented using the open-source Jieba segmentation module.

2. The BERT classification method for large-scale exploration and development based on semantic fault tolerance as described in claim 1, characterized in that, The specific steps of segmenting the categorized sentences using the open-source Jieba segmentation module include: The sentence is segmented into words using the open-source Jieba word segmentation module, resulting in sentences in word units. During the word segmentation process, a user dictionary should be added for specialized terms to improve the accuracy of word segmentation.

Images