Summary automatic generation method based on Bert-wwm-Ext model and storage medium

Abstract

The application discloses a summary automatic generation method based on a Bert-wwm-Ext model and a storage medium, and the summary automatic generation method comprises the following steps: collecting first content corresponding to a summary label under bibliographic item information and second content corresponding to an invention content label under a specification, and generating a Chinese patent data set; performing Token conversion on the Chinese patent data set; pre-training according to a Bert-wwm-Ext model; generating a first document level feature of a summary and first summary content based on the Bert-wwm-Ext model and according to a sentence vector, and evaluating the first summary content and the first content; obtaining a second document level feature in the second content according to an evaluation result, and generating second summary content according to the sentence vector corresponding to the evaluation result. The application can obtain a summary from input text, and the recall rate and F value of Rouge are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing, and particularly relates to a summary automatic generation method based on a Bert-wwm-Ext model and a storage medium. BACKGROUND

[0002] The explosive development of modern Internet has generated a large amount of data information, but the proportion of valuable information in more data information has decreased, and the effective use of text information is a big problem. The text summary technology extracts, summarizes or refines the key information in the text data information to summarize and show the main content or points of the original text information, thereby greatly improving the efficiency of users in obtaining the required information from a large amount of text data.

[0003] The summary of the Chinese patent document is a brief description and summary of the specification, and is an important part of the patent application document. In the process of conceiving and implementing the present application, the applicant found that the writing of the summary is a time-consuming and laborious work. Some automatic text generation software on the market often does not meet the requirements of the summary in the Chinese patent document, causing subsequent correction problems. The traditional way of extracting text summary by manual work requires a lot of time and effort and has a very high requirement for the professionalism of the practitioners, and does not have the possibility of large-scale application. SUMMARY

[0004] In view of the above technical problems, the present application provides a summary automatic generation method based on a Bert-wwm-Ext model and a storage medium to alleviate the problem of writing a summary of a Chinese patent document.

[0005] The present application provides a summary automatic generation method based on a Bert-wwm-Ext model, which specifically includes the following steps:

[0006] Collecting a first content corresponding to a summary label under the bibliographic item information and a second content corresponding to an invention content label under the specification to generate a Chinese patent data set;

[0007] After the Head-Tail interception of the Chinese patent data set, Token conversion is performed;

[0008] According to the Bert-wwm-Ext model, the Chinese patent data set after Token conversion is pre-trained to obtain the sentence vector between the first content and the second content;

[0009] Based on the trained Bert-wwm-Ext model, according to the sentence vector, the first document level feature for generating the summary in the first content and the second content is obtained, and according to the first association weight between the sentences corresponding to the first document level feature and the first structure weight of the sentences in the second content, the first summary content is generated, and the first summary content and the first content are evaluated;

[0010] According to the evaluation result, based on the second association weight and the second structure weight of the second content, the second document level feature in the second content is obtained, and the second summary content is generated according to the sentence vector corresponding to the evaluation result.

[0011] Optionally, in the process of generating the Chinese patent data set, the Chinese patent data set is CLTPDS data set.

[0012] Optionally, in the process of Head-Tail intercepting the Chinese patent data set, the head and tail core sentences of the second content are cut off in the ratio of 128:328, and the sentences below 4 words are cut off and discarded directly after cutting.

[0013] Optionally, in the process of Token conversion, the head of the data after Head-Tail intercepting is set as [CLS] flag bit, it is judged whether it is a summary sentence, and the tail of the sentence is set as [SEP] to separate the sentences;

[0014] Through multiple embedding calculations, input representation is input as the input of the Bert-wwm-Ext model.

[0015] Optionally, in the process of pre-training, at least one preset layer pointing to the first content is constructed according to the sentence vector, which is used to extract the first document level feature.

[0016] Optionally, in the process of generating the first summary content, the prediction score of each sentence and the loss of the whole model are obtained, wherein the loss is used to represent the binary classification entropy of the prediction score and the golden label;

[0017] According to the prediction score and the loss, the layer of the first content and the Bert-wwm-Ext model are fine-tuned.

[0018] Optionally, in the process of obtaining the prediction score, at least one composite classifier is added on the output of the Bert-wwm-Ext model and a sigmoid function is used to obtain the prediction score.

[0019] Optionally, in the process of evaluating the first summary content and the first content, the evaluation is performed by ROUGE method.

[0020] Optionally, in the process of evaluation by the ROUGE method, the first matching degree of the first content and the first summary content in the unigram, the second matching degree in the bigram, and the recall rate in the N-gram are obtained; and the quality of the first summary content is judged.

[0021] In another aspect, the application further provides a storage medium, specifically, the storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the above-mentioned summary automatic generation method based on the Bert-wwm-Ext model.

[0022] The summary automatic generation method based on the Bert-wwm-Ext model and the storage medium provided by the application can assist the writing personnel to automatically generate the summary of the Chinese patent document based on the improved Bertsum model, alleviate the writing difficulty, and reduce the correction probability of the patent application document. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application. In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0024] Figure 1 The long text patent automatic summary flowchart provided by the embodiments of the application;

[0025] Figure 2 The Head-Tail text interception schematic diagram provided by the embodiments of the application;

[0026] Figure 3 The input representation example diagram provided by the embodiments of the application;

[0027] Figure 4 The schematic diagram based on the patent text structure classification provided by the embodiments of the application;

[0028] Figure 5 The detailed result comparison diagram of the original Bertsum model and the improved Bertsum model provided by the embodiments of the application.

[0029] Figure 6 The summary automatic generation method flowchart of an embodiment of the application.

[0030] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments in conjunction with the drawings. The above drawings have shown the specific embodiments of the present application, and will be described in more detail hereinafter. These drawings and the written description are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0031] The exemplary embodiments will be described in detail herein below with reference to the drawings. In the following description, the same drawings refer to the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0032] First embodiment

[0033] A Chinese patent long text automatic abstract method based on an improved Bertsum model is improved based on the BertSum algorithm model, a sentence vector training classification framework is designed to track the generation of sentence vectors of sentences, and training and classification are performed. The framework is divided into the following five parts: relevant patent text is collected by using a python crawler, the text is preprocessed, the model is pre-trained and fine-tuned, the classification judgment of the abstract result, and the standard text evaluation. Specifically as follows:

[0034] Step 1: relevant patent text is collected by using a python crawler

[0035] Since there is no large-scale Chinese patent text data set in the Internet, the data set used in the present application is collected from the real patent data in the public patent website such as Patent Star. The data consists of patent texts of 1500 to 3000 words and corresponding text abstracts of 150 to 300 words.

[0036] The present application totally crawls about 42,000 long text abstract data for actual experiments. After data preprocessing by using python, the data is divided into 38,000 training set, 3,300 validation set and 800 test set, and is arranged into Chinese long text patent data set CLTPDS.

[0037] At the same time, the present application uses the abstract data generated by the Oracle algorithm to obtain 24k short text Chinese abstracts. The data is only used for the feasibility test of the algorithm.

[0038] Step 2: text preprocessing

[0039] Based on the Chinese long text patent data set CLTPDS obtained in step 1, the text structure and scale characteristics of the patent article need to be processed.

[0040] Since the pre-trained model is based on the Bert model, and the Bert consumes the computing resources and time which grows in square level with the length of token, it cannot process too long token, and currently only supports 512 tokens, and too long token is also easy to overflow, so when using Bert to process long text, a clever method needs to be designed to solve this problem.

[0041] Currently, the industry mainly adopts three methods of Head-Tail interception, segmentation fusion and Sliding Window sliding window to process long text. Based on the characteristics of patent text, the Head-Tail method with low computing resource consumption is adopted to process the obtained data.

[0042] After the data is preliminarily processed, the input representation is performed.

[0043] The input representation can explicitly represent a single text sentence or a pair of text sentences in a word block sequence. For a given word block, its input representation is constructed by summing the word block embedding, segment embedding and position embedding of the corresponding word block.

[0044] Using the learned position embedding, the supported sequence length is up to 512 word blocks.

[0045] The first word block of each sequence is always a special classification embedding ([CLS]). The final hidden state of the word block (i.e. the Transformer output) is used as the aggregated sequence representation for classification tasks. For non-classification tasks, this vector will be ignored.

[0046] For patent text processing, input representation processing is performed after text segmentation (the processing is shown in Figure 3 ). The same text will be processed at the same time, and sentence one and sentence two are only examples to illustrate how to distinguish multiple sentences. Figure 3 Sentence one is processed in the name of sentence A, sentence two is processed in the name of sentence B, and sentence three is processed in the name of sentence A again, and so on. Among them, sentence A and sentence B are only a code, representing different sentences, and have no specific meaning.

[0047] If it is a single sentence input, only sentence A embedding is used.

[0048] Step 3: Pre-training and fine-tuning of the model

[0049] The pre-training model is BERT-wwm-ext, which is a Chinese pre-training language model released by the Harbin Institute of Technology and Xunfei Joint Laboratory, and is an upgraded version of BERT-wwm.

[0050] The pre-training model is an upgraded version of BERT released by Google on May 31, 2019, which mainly changes the training sample generation strategy in the original pre-training stage. The original WordPiece-based word segmentation method will divide a complete word into several subwords, and these divided subwords will be randomly masked when generating training samples.

[0051] In full word Mask, if a part of the WordPiece subword of a complete word is masked, other parts belonging to the word will also be masked, that is, full word Mask.

[0052] It should be noted that mask here refers to the broad sense of mask (replaced by [MASK]; keep the original word; randomly replaced by another word), and is not limited to the case of replacing a word with a [MASK] tag.

[0053] In the fine-tuning process, the same architecture is used for pre-training and fine-tuning except for the output layer. The same pre-training model parameters are used to initialize the model for different downstream tasks. In the fine-tuning process, all parameters are fine-tuned. The present application is fine-tuned for the direction of extracting summaries of BERT.

[0054] After obtaining the sentence vector from the output of the pre-training model, several summary-specific layers will be constructed on it to capture document-level features for extracting summaries.

[0055] For each sentence T i , the final prediction score is calculated The loss of the entire model is The binary classification entropy of the gold label Y i . These summary layers will be fine-tuned together with BERT.

[0056] A composite classifier is added to the output of the pre-training model and a sigmoid function is used to obtain the prediction score:

[0057]

[0058] W a = W o + W s

[0059] where σ is the sigmoid function, W o is the internal sentence association weight of the output sentence vector, and W sThe structural weight of a sentence in an article is determined by the length of the sentence and the position of the sentence in the article.

[0060] Step 4: Standard text evaluation

[0061] The most commonly used evaluation method in text summarization tasks is ROUGE (Recall-Oriented Understudy for Gisting Evaluation), which supports three evaluations: Rouge1, Rouge2, and Rouge-L. Rouge1 evaluates the matching degree of one-word grammar between the standard text and the generated text, Rouge2 evaluates the matching degree of two-word grammar, and Rouge-L calculates the longest common word length matching degree between the standard text and the generated text. The invention selects the most discriminative Rouge-L as the main reference score.

[0062] The calculation method is as follows:

[0063]

[0064]

[0065]

[0066] where S is the human summary, C is the machine summary, R LCS represents the recall rate, P LCS represents the precision rate, and F LCS The result of the F value of Rouge-L is the F value of Rouge-L.

[0067] The present application studies the writing structure of patent articles, the size of the organization, and explores the rules of the location and frequency of the summary in the original text, and then establishes an automatic summary extraction model considering the characteristics of patent text and its summary. The present application believes that BertSum has a lot of room for improving the performance of extracting summaries by virtue of its pre-training method on a large Chinese data set and its powerful architecture for learning complex features of summaries.

[0068] Figure 1 The long text patent automatic summary flowchart provided by the embodiments of the present application.

[0069] Please refer to Figure 1 Alternatively, the automatic summary generation process steps are as follows:

[0070] Step 1: Use python crawler to collect relevant patent text.

[0071] To better adapt to the needs of real patent summary extraction, the model established by the present technology needs a large-scale real Chinese patent data set for training.

[0072] Since there is no large-scale Chinese patent text data set in the Internet, the data set used in the present application is collected from real patent data from the public patent website such as PatSnap.

[0073] The present application uses python to access the PatSnap retrieval website to crawl public patent data. After entering the website, the patent content in each field is crawled in turn. The crawled data includes the content under the summary label of the registration item information and the invention content label under the specification. Finally, the patent data composed of patent text of 1500 to 3000 words and corresponding text summary of 150 to 300 words is obtained.

[0074] The present application totally crawls about 42,000 super-long text summary data for actual experiment. After data preprocessing by python, the data is divided into 38,000 training set, 3,300 validation set and 800 test set, and is arranged into Chinese long text patent data set CLTPDS.

[0075] At the same time, the present application uses the summary data generated by Oracle algorithm to obtain 24k short text Chinese summary. This data is only used for feasibility test of the algorithm.

[0076] The following is the comparison between Chinese long text patent data set CLTPDS and TTNews of classic long text summary data set.

[0077] Table 1 comparison of text length of data set

[0078]

[0079] As can be seen from Table 1, compared with the TTNews data set, the average length of the CLTODS data set is 3 times that of the former, reaching more than 1500, and the average length of the summary text is 6 times that of the former, reaching more than 150. Therefore, the CLTODS data set can more truly reflect the requirements of current patent text and its summary, and can better adapt to the goal to be achieved by the present application.

[0080] Step 2: Preprocessing of text (data cleaning, Head-Tail truncation, Token conversion).

[0081] Figure 2 The Head-Tail text truncation schematic diagram provided for the embodiments of the present application.

[0082] Please refer to Figure 2A complete Chinese patent specification includes a title page, claims, a specification, and drawings. The main narrative of the patent abstract is in the invention content column of the specification, and the patent invention text is generally more than 1500 words, and the patent abstract text is 100 to 300 words, so the text structure and scale characteristics of the patent article need to be processed.

[0083] The data collected when matching the invention content of the patent specification and the corresponding abstract is considered. After removing errors and repetitions from the collected data, the data is sorted into a Chinese long text patent data set CLTPDS. Considering that pre-trained models are all based on the Bert model, and the Bert consumes computing resources and time with the length of the token growing in a square level, it cannot handle too long tokens. Currently, the longest supports 512 tokens, and too long tokens are also prone to memory overflow. Therefore, when using Bert to process long text patents, a clever method needs to be designed to solve this problem.

[0084] After the data is preliminarily processed, the input representation is input. Figure 3 An example of the input representation provided by the embodiments of the present application is shown in the figure.

[0085] Please refer to Figure 3 After obtaining the data of the data set, the text processing required by the model is performed, including Head-tail processing, text segmentation, etc.

[0086] After segmentation, [CLS] and [SEP] markers are inserted before and after each sentence in the document. The classification token ([CLS]) is used to aggregate the entire sequence of token representations. The [CLS] token before each sentence is the sentence vector representation of the sentence after entering the model. At the same time, the division token ([SEP]) is inserted after each sentence in the sequence of tokens to separate different sentence tokens. This allows the model to accurately distinguish the range of each sentence. After the above processing, the input document is obtained.

[0087] If the input sequence only contains one sentence, there is no [SEP] and the token after it.

[0088] Token embedding: convert tokens into corresponding representations, and calculate the corresponding representation of each word separately. The [CLS] record contains the representation information set of the sequence.

[0089] Separate embedding: add a learnable Segment Embeddings to each token representation to indicate whether it belongs to sentence A or sentence B (not limited to two sentences).

[0090] Position Embeddings: Add a Position Embeddings to token representation to distinguish the same word in different positions of the sequence, reduce the influence of polysemy.

[0091] At this time, the constructed input representation can be input into the pre-trained model for the next step.

[0092] The input representation can explicitly represent a single text sentence or a pair of text sentences in a word block sequence. For a given word block, its input representation is constructed by summing the word block embedding, segment embedding and position embedding of the corresponding word block.

[0093] Currently, long text research mainly adopts Head-Tail cutting, segmentation fusion and Sliding Window sliding window three methods to process long text. Based on the characteristics of the core content of patent text concentrated in the head and tail of the text, the Head-Tail method focusing on the head and tail core sentence and low resource consumption is adopted for processing. In the processing process, the head and tail core sentences of the invention content in the patent specification are cut off at a ratio of 128:328, and the sentences below 4 words are discarded after cutting and dividing.

[0094] The experimental results prove that the addition of Head-Tail can significantly improve the quality of the summary under the relatively lowest complexity.

[0095] The input data is tokenized, the [CLS] flag is set at the beginning of the input data sentence, it is judged whether it is a summary sentence, and the [SEP] separator is set at the end of the sentence.

[0096] Multiple embedding calculations are performed to obtain sentence vectors containing inter-sentence and intra-sentence features as inputs to the pre-trained model.

[0097] Step 3: Pre-training and fine-tuning of the model.

[0098] The pre-trained model used in the present application is the Bert-wwm-Ext model, which is a Chinese pre-trained language model released by the Harbin Institute of Technology and the Xunfei Joint Laboratory. Compared with other pre-trained Bert models, the pre-training data set of this model is increased, and the number of times reaches 5.4B; the training step is increased, and the first training stage is 1M steps, and the second training stage is 400K steps, which has better performance in the field of Chinese text summary.

[0099] In order to achieve the goal of generating excellent long text summary, the Bertsum model is improved to additionally consider the influence of sentence length, sentence text structure and other factors on summary judgment. After obtaining the sentence vector from the output of the pre-trained model, several specific summary layers are constructed thereon to capture document-level features for extracting summaries.

[0100] Figure 4The schematic diagram based on patent text structure classification provided for the embodiments of the present application.

[0101] Please refer to Figure 4 After obtaining the sentence vector of each sentence through the pre-training model, several specific layers for abstract are constructed to capture document-level features for extracting abstract, as follows:

[0102] Ti is the [CLS] label of the input representation, and the feature information of the sequence is calculated after the pre-training model processing. The information is obtained by the multiple self-attention mechanism of the pre-training model, including the relationship between the sequence and other sequences and the relationship between the words in the sequence.

[0103] For each sentence T i , the final prediction score is calculated The loss of the whole model is the binary classification entropy of the gold label Y i . These summary layers will be fine-tuned with BERT.

[0104] A composite classifier is added to the output of the pre-training model, and a sigmoid function is used to obtain the prediction score:

[0105]

[0106] W a = W o + W s

[0107] Where σ is the sigmoid function, W o is the internal sentence association weight of the output sentence vector, and W s is the structure weight of the sentence in the article, including the length of the sentence and the weight determined by the position of the sentence in the article.

[0108] Step 4: Standard text evaluation

[0109] The most commonly used evaluation method in the text abstract task is ROUGE (Recall-Oriented Understudy for Gisting Evaluation), which supports Rouge1, Rouge2, and Rouge-L three kinds of evaluation. Rouge1 is to evaluate the matching degree of the standard text and the generated text in the one-gram, Rouge2 is to evaluate the two-gram matching degree, and ROUGE-N mainly calculates the recall rate of N-gram. For N-gram, the ROUGE-N score can be calculated, and the calculation formula is as follows:

[0110]

[0111] Rouge-L is the longest common subsequence score between the standard text and the generated text. The most discriminative Rouge-L is selected as the main reference score in this invention.

[0112] The calculation method is as follows:

[0113]

[0114]

[0115]

[0116] where S is the human summary, C is the machine summary, R LCS represents the recall, P LCS represents the precision, and F LCS The result of F is the F value score of Rouge-L.

[0117] Chinese patent text summary is a new field, and there is less related work. At the same time, since the generated summary result is more than 100 words, a higher evaluation will be obtained in the Rouge calculation, which is different from the traditional Rouge baseline, and cannot be compared with the existing work. Therefore, the existing model is implemented on the long text summary dataset in this application, and the comparison is as follows. All models use the Chinese long text patent dataset CLTPDS.

[0118] 1) TextRank: TextRank algorithm is an algorithm for sorting words and sentences by constructing a topological structure graph of text. First, the patent text is divided into several component sentences to construct a node connection graph, then the similarity between sentences is calculated to generate the weight value of the edge, then the TextRank value of the sentence is calculated by iterative calculation, and finally the top three sentences with the highest TextRank value are extracted to combine into a text summary according to the original text order, so as to obtain a relatively smooth logical order summary.

[0119] 2) Transformer: proposed by Vaswani et al. in 2017, the Transformer model is a completely multi-head attention mechanism based model. The baseline used in this application is 6 layers, the hidden layer size is 512, and the feed filter size is 2048. The top three scores of the generated results are combined to form a summary.

[0120] 3) Improved Bertsum-a: Bertsum model with Head-Tail technology, the top three scores of the generated results are combined to form a summary.

[0121] 4) Improved Bertsum-b: the Bertsum model with the addition of a classifier based on the structure of the patent text, the results calculated by the improved Bertsum model are obtained after passing through the classifier.

[0122] 5) Improved Bertsum: the Head-Tail technology and the classifier based on the structure of the patent text are added at the same time, the text is processed by the Head-Tail technology, and the results are calculated based on the classic Bertsum model, and the results are generated by the classifier based on the structure of the patent.

[0123] Table 2: Evaluation table of results of each model

[0124]

[0125] As shown in Table 2, Bertsum has a decisive advantage over other general models in the field of patent abstracts. The improved Bertsum has achieved the best results in the three indicators. Compared with the original Bertsum, the Bertsum with the addition of the Head-Tail technology and the classifier based on the structure of the patent text has a significant improvement in performance, and the improved Bertsum model with the addition of the classifier based on the structure of the patent text on the basis of the former has obtained extremely excellent results.

[0126] Figure 5 The detailed result comparison chart of the original Bertsum model and the improved Bertsum model provided for the embodiments of the present application is shown in the following figure.

[0127] As shown in the following figure, Figure 5 from the overall distribution, the results generated by the Bertsum model are more in the interval of 0.5 before, and the improved Bertsum has a significant advantage when the score reaches 0.6 and above. That is, the improved Bertsum model not only reduces low-quality text abstracts, but also generates more high-quality text abstracts, and the model has stability. From the specific numerical value, the ROUGE-L expectation of Bertsum is 60.93, and the ROUGE-L expectation of the improved Bertsum model is 66.78, which has an improvement of 9.6 percent in this test alone. In general, the improved Bertsum model has a significant improvement in expectation value compared with the original model, and the overall distribution is more biased towards high scores, and the improved Bertsum model can stably generate high-quality patent text abstracts compared with the Bertsum model.

[0128] Second embodiment

[0129] The present application also provides a summary automatic generation method based on a Bert-wwm-Ext model, Figure 6 The flow chart of the summary automatic generation method of an embodiment of the present application is shown in the following figure.

[0130] Reference is made to Figure 6 In an embodiment, the automatic abstract generation method comprises the following steps:

[0131] S10: Collect the first content corresponding to the abstract label under the bibliographic item information and the second content corresponding to the invention content label under the specification, and generate a Chinese patent data set;

[0132] S20: After Head-Tail cutting of the Chinese patent data set, Token conversion is performed;

[0133] S30: According to the Bert-wwm-Ext model, the Token-converted Chinese patent data set is pre-trained to obtain the sentence vector between the first content and the second content;

[0134] S40: Based on the Bert-wwm-Ext model, the first document-level feature for generating the abstract is obtained from the sentence vector in the first content and the second content, and the first abstract content is generated according to the first association weight between the sentences corresponding to the first document-level feature and the first structure weight of the sentences in the second content. The first abstract content is evaluated with the first content;

[0135] S50: According to the evaluation result, the second document-level feature in the second content is obtained based on the second association weight and the second structure weight of the second content, and the second abstract content is generated according to the sentence vector corresponding to the evaluation result.

[0136] Optionally, the abstract score generated during model training is fed back to the model for fine-tuning. After the model training is completed, the input text can obtain the abstract using the model.

[0137] Optionally, in the process of generating the Chinese patent data set, the Chinese patent data set is a CLTPDS data set.

[0138] Illustratively, CLTPDA is the final generated data set.

[0139] Optionally, in the process of Head-Tail cutting of the Chinese patent data set, the head and tail core sentences of the second content are cut at a ratio of 128:328, and the cut sentences are discarded directly if the number of words is less than 4.

[0140] Optionally, in the process of Token conversion, the beginning of the data after Head-Tail cutting is set as a [CLS] flag bit, which is used to determine whether it is an abstract sentence, and the end of the sentence is set as a [SEP] separator to separate the sentences.

[0141] Through multiple embedding calculations, the input representation is used as the input of the Bert-wwm-Ext model.

[0142] Optionally, in the process of pre-training, at least one preset layer pointing to the first content is constructed according to the sentence vector, for extracting the first document-level feature.

[0143] Optionally, in the process of generating the first summary content, a prediction score of each sentence and a loss of the entire model are obtained, wherein the loss is used to represent the binary classification entropy of the prediction score and the golden label.

[0144] According to the prediction score and the loss, the layer of the first content and the Bert-wwm-Ext model are fine-tuned.

[0145] Optionally, in the process of obtaining the prediction score, the prediction score is obtained by adding at least one composite classifier on the output of the Bert-wwm-Ext model and using a sigmoid function.

[0146] Optionally, in the process of evaluating the first summary content and the first content, the evaluation is performed by the ROUGE method.

[0147] Optionally, in the process of evaluating by the ROUGE method, the quality of the first summary content is judged by obtaining a first matching degree in the unigram, a second matching degree in the bigram and a recall rate in the N-gram of the first content and the first summary content.

[0148] In another aspect, the present application also provides a storage medium, specifically, a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the above-mentioned summary automatic generation method based on the Bert-wwm-Ext model.

[0149] The summary automatic generation method based on the Bert-wwm-Ext model and the storage medium provided by the present application can assist the writing personnel to automatically generate the summary of the Chinese patent document based on the improved Bertsum model, alleviate the writing difficulty of part of the writing personnel, and reduce the correction rate of the patent document.

[0150] Third embodiment

[0151] In another aspect, the present application also provides a storage medium, specifically, a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the above-mentioned encoding method.

[0152] The application is based on the improved Bertsum model BERT-wwm-ext, and the improvement is that the Head-tail method is introduced to process the excessively long input text. The BERT-wwm-ext model is more suitable for Chinese text and improves the performance of calculating sentence vectors. The classifier is changed from the original simple linear classifier to the classifier based on the structure of the patent text, and can adaptively and dynamically select the appropriate summary sentence to make the final summary length more suitable for the needs of the patent regulations. As described above, the Bertsum model based on the automatic summary of the patent text in the application uses the Chinese long text patent dataset CLTPDS, introduces the Head-Tail technology based on the Bertsum model and optimizes the classifier, solves the problem of how to extract and generate high-quality long text from the super-long patent text summary.

[0153] It should be noted that in this paper, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the sentence "includes a" does not exclude the presence of another identical element in the process, method, article or device including the element, in addition, the components, features, elements with the same name in different embodiments of the application may have the same meaning or different meanings, and the specific meaning thereof should be determined in combination with the explanation in the specific embodiment or the context in the specific embodiment.

[0154] It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this paper, the first information can also be called the second information, and similarly, the second information can also be called the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining". Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include plural forms, unless the context indicates the contrary.

[0155] It should be noted that in this paper, step codes such as S10, S20, etc. are used to more clearly and briefly express the corresponding content, and do not constitute substantial restrictions on the order. Those skilled in the art may perform S20 before S10 in specific implementation, etc., but these should be within the protection scope of the application.

[0156] In the embodiments of the intelligent terminal and the computer readable storage medium provided in the application, all technical features of any of the above method embodiments for automatically generating summaries based on the Bert-wwm-Ext model can be included, and the description and explanation content is basically the same as that of the above method embodiments, which will not be repeated here.

[0157] The embodiments of the application also provide a computer program product, which comprises computer program code, and when the computer program code is run on a computer, the computer is caused to execute the method in various possible embodiments as above.

[0158] The embodiments of the application also provide a chip, which comprises a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the device installed with the chip executes the method in various possible embodiments as above.

[0159] The technical features of the technical solutions of the application can be combined arbitrarily, and in order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the range recited in the application.

[0160] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be realized by means of software and necessary general hardware platform, of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as above, and includes a plurality of instructions to make a terminal device (which can be a mobile phone, computer, server, controlled terminal, or network device, etc.) execute the method of each embodiment of the application.

[0161] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, storage disk, magnetic tape), optical media (for example, DVD), or semiconductor media (for example, solid state disk (SSD)) and the like.

[0162] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for automatically generating an abstract based on a Bert-wwm-Ext model, characterized in that, The method comprises the following steps: Collecting the first content corresponding to the summary label under the bibliographic item information and the second content corresponding to the invention content label under the specification to generate a Chinese patent data set; After Head-Tail cutting of the Chinese patent data set, Token conversion is performed; According to the Bert-wwm-Ext model, the sentence vector between the first content and the second content is obtained by pre-training the Token converted Chinese patent data set; Based on the trained Bert-wwm-Ext model, the first document level feature for generating the summary is obtained from the sentence vector, and the first summary content is generated according to the first association weight between the sentences corresponding to the first document level feature and the first structure weight of the sentences in the second content, and the first content is evaluated; According to the evaluation result, the second document level feature in the second content is obtained based on the second association weight and the second structure weight of the second content, and the second summary content is generated according to the sentence vector corresponding to the evaluation result.

2. The automatic summary generation method based on the Bert-wwm-Ext model according to claim 1, wherein: In the process of generating the Chinese patent data set, the Chinese patent data set is the CLTPDS data set.

3. The automatic summary generation method based on the Bert-wwm-Ext model according to claim 2, wherein: In the process of Head-Tail cutting of the Chinese patent data set, the head and tail core sentences of the second content are cut at a ratio of 128:328, and the sentences below 4 words are discarded after cutting and dividing.

4. The automatic summary generation method based on the Bert-wwm-Ext model according to claim 3, wherein: In the process of Token conversion, the [CLS] flag bit is set at the beginning of the data after Head-Tail cutting to determine whether it is a summary sentence, and the [SEP] separator is set at the end of the sentence to separate the sentences; Through multiple embedding calculations, input representation is obtained as the input of the Bert-wwm-Ext model.

5. The automatic summary generation method based on the Bert-wwm-Ext model according to claim 4, wherein: In the process of pre-training, at least one preset layer pointing to the first content is constructed according to the sentence vector to extract the first document level feature.

6. The automatic summary generation method based on the Bert-wwm-Ext model according to claim 5, wherein: In the process of generating the first summary content, the prediction score of each sentence and the loss of the entire model are obtained, wherein the loss is used to represent the binary classification entropy of the prediction score and the golden label; According to the prediction score and the loss, the layers of the first content and the Bert-wwm-Ext model are fine-tuned.

7. The method of claim 6, wherein the prediction score is obtained by adding at least one composite classifier on the output of the Bert-wwm-Ext model and using a sigmoid function.

8. The method of claim 7, wherein the first summary content is evaluated against the first content by using a ROUGE method.

9. The method of claim 8, wherein the quality of the first summary content is determined by obtaining a first matching degree in a unigram, a second matching degree in a bigram, and a recall rate in an N-gram of the first content and the first summary content.

10. A computer program product, comprising a computer readable storage medium having stored thereon computer program instructions that, when executed at a processor, implement the method of any of claims 1-9. ​ ​ 10. A storage medium, characterized by ​

Images