Sentence simplification method, apparatus, device, and computer-readable storage medium

Abstract

A sentence simplification method is disclosed, comprising the steps of: obtaining a sentence to be simplified; marking the sentence to be simplified to obtain a plurality of first-granularity sentence elements; determining a syntactic embedding of the sentence to be simplified, the syntactic embedding comprising a vector indicating a dependency relationship of each of the first-granularity sentence elements; predicting a topic role of each of the first-granularity sentence elements based at least on the syntactic embedding, the topic role indicating a relationship between the first-granularity sentence element and a topic of the sentence to be simplified; and forming a simplified sentence using the first-granularity sentence elements predicted to be the expected topic roles. Corresponding devices, equipment, and computer-readable storage media are also disclosed.

Description

Technical Field

[0001] The present application relates to sentence simplification, and more particularly, to a sentence simplification method, apparatus, device, and computer-readable storage medium. Background Art

[0002] With the rapid development of the Internet, content data such as text, voice, images and videos have entered an era of big explosion. In an era with such a huge amount of data, the speed of absorbing information in the content has become very important. Compared with text and images, audio and video data have added a time dimension, so it is more time-consuming to understand. Although audio and video can be converted into text content through certain technologies, this also brings great challenges to the reading speed of the text. A technology is needed that can help users quickly understand the text content and retrieve targets in massive data.

[0003] Sentence simplification refers to deleting redundant words or phrases in a sentence, and retaining concise phrases that can sufficiently represent the core information of the sentence. In actual business scenarios, people often hope to obtain the overall information of a sentence by browsing phrase-level topics. Summary of the invention

[0004] In view of this, the present disclosure provides a sentence simplification method, a sentence simplification apparatus, a computing device, a computer-readable storage medium, and a computer program product, which can alleviate, mitigate, or even eliminate the above-mentioned problems.

[0005] According to a first aspect of the present invention, a sentence simplification method is provided, comprising the steps of: obtaining a sentence to be simplified; marking the sentence to be simplified to obtain a plurality of first-granularity sentence elements; determining a syntactic embedding of the sentence to be simplified, the syntactic embedding comprising a vector indicating a dependency relationship of each of the first-granularity sentence elements; predicting a topic role of each first-granularity sentence element based at least on the syntactic embedding, the topic role indicating a relationship between the first-granularity sentence element and a topic of the sentence to be simplified; and forming a simplified sentence using the first-granularity sentence elements predicted to have the desired topic role.

[0006] According to one embodiment, determining the syntactic embedding of the sentence to be simplified further includes: based on the second granularity, performing dependency syntactic analysis on the sentence to be simplified to obtain dependency relationships of second granularity sentence elements, wherein each of the second granularity sentence elements includes one or more first granularity sentence elements; determining the dependency relationship of each second granularity sentence element as the dependency relationship of each first granularity sentence element included in the second granularity sentence element, and generating a corresponding vector for each first granularity sentence element based on the dependency relationship, thereby obtaining the syntactic embedding.

[0007] According to one embodiment, the method also includes: determining the tag embedding of the sentence to be simplified, the tag embedding includes the word vector of each first-granularity sentence element; determining the fragment embedding of the sentence to be simplified, the fragment embedding includes a vector indicating which fragment each first-granularity sentence element belongs to, the fragment including a sentence or a component of a sentence; determining the position embedding of the sentence to be simplified, the position embedding includes a vector indicating the position of each first-granularity sentence element; wherein the prediction further includes: predicting the topic role of each first-granularity sentence element based at least on the syntactic embedding, the tag embedding, the fragment embedding and the position embedding.

[0008] According to one embodiment, the prediction further includes: adding the syntactic embedding, the tag embedding, the fragment embedding and the position embedding to obtain the sum of the embeddings, and predicting the topic role of each first-granularity sentence element based on the sum of the embeddings.

[0009] According to one embodiment, the sentence is a Chinese sentence, and the first granularity is a word.

[0010] According to one embodiment, the sentence is a Chinese sentence, and the second granularity is a word.

[0011] According to one embodiment, the prediction further includes: taking the syntactic embedding as at least a part of the input embedding of a trained natural language processing model, predicting the topic role of each first-granularity sentence element via the natural language processing model, wherein the natural language processing model includes any one of the following: BERT model, ERNIE model or RoBERT model According to one embodiment, the prediction further includes: predicting the probability of each first-granularity sentence element as each topic role; and determining the topic role with the largest predicted probability for each first-granularity sentence element as the topic role of the first-granularity sentence element.

[0012] According to one embodiment, the topic role includes: a first first-granularity sentence element of the topic, an intermediate first-granularity sentence element of the topic, and an off-topic element.

[0013] According to one embodiment, the expected topic role includes: a first first-granularity sentence element of the topic and an intermediate first-granularity sentence element of the topic.

[0014] According to a second aspect of the present invention, a sentence simplification device is provided, comprising: a sentence acquisition module, configured to acquire a sentence to be simplified; a labeling module, configured to label the sentence to be simplified to obtain a plurality of first-granularity sentence elements; a syntactic embedding determination module, configured to determine the syntactic embedding of the sentence to be simplified, the syntactic embedding comprising a vector indicating a dependency relationship of each of the first-granularity sentence elements; a prediction module, configured to predict a topic role of each first-granularity sentence element based at least on the syntactic embedding, the topic role indicating a relationship between the first-granularity sentence element and a topic of the sentence to be simplified; and a formation module, configured to form a simplified sentence using the first-granularity sentence elements predicted to have the desired topic role.

[0015] According to a third aspect of the present invention, there is provided a statement simplification device, comprising: a processor; and a memory configured to store thereon computer executable instructions, which, when executed in the processor, enable the processor to implement the method according to the first aspect of the present invention.

[0016] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein when the instructions are executed by a computing device, the computing device implements the method according to the first aspect of the present invention.

[0017] According to an embodiment of the present invention, the introduction of syntax embedding is helpful to take syntax factors into consideration when simplifying sentences, which can help improve the accuracy and readability of sentence simplification. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 An example application scenario in which sentence simplification according to some embodiments of the present disclosure can be applied is schematically illustrated.

[0020] Figure 2 A schematic diagram of a model on which the sentence simplification method provided by an embodiment of the present disclosure is based is illustrated.

[0021] Figure 3 The diagram shows the dependency relationships of statements.

[0022] Figure 4a The figure illustrates the method of generating input embedding in the related art.

[0023] Figure 4b The diagram illustrates how input embedding is generated according to an embodiment of the present disclosure.

[0024] Figure 5 The diagram shows the model structure of Transformer.

[0025] Figure 6 The flowchart of the sentence simplification method provided by the embodiment of the present disclosure is illustrated.

[0026] Figure 7 The diagram illustrates a method for generating syntactic embedding provided by an embodiment of the present disclosure.

[0027] Figure 8 Another flowchart of the sentence simplification method provided by the embodiment of the present disclosure is illustrated.

[0028] Fig. 9 The figure illustrates the topic role prediction method provided by the embodiment of the present disclosure.

[0029] Fig.10 The figure shows a block diagram of a sentence simplification device provided by an embodiment of the present invention;

[0030] Fig.11 The figure shows a schematic diagram of a hardware implementation environment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] For the sentence simplification method, apparatus, device, and computer-readable storage medium provided in the embodiments of the present application, in order to make the purpose, technical solution and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0032] The terms “each”, “plurality” and the like used in this application include two or more than two, and each refers to each of the corresponding plurality.

[0033] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0034] Artificial Intelligence (AI) mentioned in this application refers to theories, methods, technologies and application systems that use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0035] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0036] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0037] Before describing the embodiments of the present disclosure in detail, some related concepts are first explained:

[0038] Embedding: Formally speaking, Embedding is to "represent" an object with a low-dimensional dense vector. The object here can be a word, an item, or a node in a network relationship. The word "representation" means that Embedding can express certain characteristics of the corresponding object, and the distance between vectors reflects the similarity between objects.

[0039] Dependency relations: Grammatical relations between words in a sentence. Here are some examples of dependency relations:

[0040]

[0041]

[0042] Dependency syntactic analysis: Explains the syntactic structure by analyzing the dependency relationship between components within a language unit, and advocates that the core verb in a sentence is the central component that controls other components. However, it itself is not controlled by any other component, and all controlled components are subordinate to the controller in some way. There is a direct dependency relationship between words, forming a dependency pair, one of which is the core word, also called the dominant word, and the other is the modifier, also called the subordinate word. The dependency relationship is represented by a directed arc, called a dependency arc. The direction of the dependency arc is from the subordinate word to the dominant word, and of course the reverse is also possible. It can be expressed uniformly according to personal habits, such as Figure 3 shown.

[0043] Topic role: Topic role indicates the relationship between the elements in the sentence to be simplified and the topic of the sentence to be simplified. For example, the topic role may include: the first first-granularity sentence element of the topic, the middle first-granularity sentence element of the topic, and the off-topic element.

[0044] Word embedding is a general term for a set of language modeling and feature learning techniques in Natural Language Processing (NLP), in which words or phrases from a vocabulary are mapped to vectors of real numbers. Conceptually, it involves a mathematical mapping from a one-dimensional space for each word to a continuous vector space with lower dimensions. Methods for generating such mappings include neural networks, dimensionality reduction of word co-occurrence matrices, probabilistic models, interpretable knowledge base methods, and explicit representation of terms.

[0045] Figure 1 An example application scenario 100 is schematically shown in which sentence simplification according to some embodiments of the present disclosure can be applied. Figure 1 As shown, the scene 100 may include a server 110 , terminal devices 130 , 150 , 170 , and optionally a database device 120 for storing texts. These servers or devices may communicate with each other via a network 190 .

[0046] Exemplarily, the terminal 130 of the user 140, the terminal 150 of the user 160, and the terminal 170 of the user 180 may upload content files to the server 110 via the network 190, including original text files, or text files converted from media files such as video files, short video files, and audio files, or multimedia files such as video files, short video files, and audio files, and converted into text files by the server. The uploaded content files may be content files stored on any one or more of the terminals 130, 150, and 170, or may be content files read from an external storage device or recorded or input by the terminals 130, 150, and 170.

[0047] The server 110 may receive the uploaded text file, or media file such as a video file, a short video file, an audio file, etc. and convert it into a text file, and perform sentence simplification processing according to the methods provided in various embodiments of the present disclosure. The processed file may be stored in a storage device of the server 110, or may be stored in an external database device 120, or may also be directly sent back to the uploading terminal or to another terminal via the network 190.

[0048] In the present disclosure, the server 110 may be a single server or a server cluster, on which an application for executing the statement simplification method according to various embodiments of the present disclosure may be run and related data may be stored. Optionally, the server 110 may also run other applications and store other data. For example, the server 110 may include multiple virtual hosts to run different applications and provide different services.

[0049] In the present disclosure, the terminals 130, 150 and 170 may be various types of devices, such as mobile phones, tablet computers, laptop computers, wearable devices such as smart watches, vehicle-mounted devices, etc. Optionally, the terminals 130, 150 and 170 may be the same or different types of terminals.

[0050] In the present disclosure, the database device 120 can be regarded as an electronic file cabinet, that is, a place to store electronic files, and users can add, query, update, delete, etc. data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program.

[0051] In addition, in the present disclosure, the network 190 may be a wired network connected via a cable, optical fiber, etc., or a wireless network such as 2G, 3G, 4G, 5G, Wi-Fi, Bluetooth, ZigBee, Li-Fi, etc.

[0052] In the related art, there are several ways to simplify sentences: keyword method, syntactic analysis method, and generative method. The keyword method segmented the sentence, extracted keywords through a keyword extraction algorithm, and then combined them together to represent the meaning of the sentence. The syntactic analysis method parses the syntactic structure of the sentence, obtains the corresponding dependency roles, and then combines the words corresponding to some dependency roles based on empirical rules. The generative method uses a deep neural network to first use an encoder to encode the original text into a hidden state of an intermediate layer, and then uses a decoder to decode the hidden state into another text.

[0053] For the above keyword method, on the one hand, the extraction of keywords depends on the existing word segmentation dictionary, which may lead to inaccurate word segmentation, for example, it is impossible to cover new words in time, it is impossible to cover professional field words, etc. On the other hand, the combination of keywords may be incoherent, and the readability is poor. For the above syntactic parsing method, the dependency relationship of each sentence can be obtained through syntactic parsing, but the same set of combination templates cannot be reused when extracting the corresponding role combination, and it often takes a lot of time to define the combination rules. For the above generative method, on the one hand, the quality and diversity of the training data set have a great impact on the results. It is difficult to have available corpus for specific fields. On the other hand, the quality of generation is poor, and there will be incomplete words, incoherence, repeated and single content, etc., and the readability is poor.

[0054] Based on the above considerations, the inventors have proposed a new method for sentence simplification. Various embodiments of the proposed sentence simplification method will be described in detail below in conjunction with specific drawings. It should be understood that in the following detailed description, numerous non-limiting specific details are set forth to help understand the subject matter presented herein. However, it is obvious to those of ordinary skill in the art that various alternatives can be used.

[0055] Figure 2 The schematic diagram of the model on which the sentence simplification method provided by an embodiment of the present invention is based is illustrated. It should be noted that the model referred to herein refers to a model in the field of artificial intelligence, unless explicitly stated otherwise. It should also be noted that the content or media referred to in the present invention includes but is not limited to video, audio, FLASH, animation, etc., which include text, audio and / or images, etc. Taking video as an example, it includes texts such as subtitles, barrage, comments, and also has an existing title in text form. For videos captured from long videos, there are also existing titles of long videos in text form, all of which can be extracted and converted into text, and the audio included in the video or audio, such as dialogue, lyrics, etc. can all be converted into text. The specific method of converting or extracting into text can be implemented by a technician according to the technical means of the relevant field, and it is not intended to be within the scope of discussion in this disclosure.

[0056] according to Figure 2 , the sentences in the text, i.e. the sentences to be simplified, are first tokenized after being input into the model to obtain a plurality of first-granularity sentence elements Tok1, Tok2, ..., Tokn, such as a plurality of words, where the first-granularity sentence elements are also called tokens. When tokenizing, the beginning [CLS] and the end [SEP] of the sentence are also marked, which are also used as first-granularity sentence elements. A special tokenizer can be used to tokenize the input sentence sequence.

[0057] In the example embodiment, the lengths of the sentences are not necessarily equal. This problem can be solved by using a fixed-length input sequence (truncation if long and padding if short). First, a seq_length hyperparameter needs to be set, and then it is determined whether the length of the entire sentence sequence exceeds the limit. If it exceeds the limit, the last part that exceeds the limit is truncated for a single sentence, and the end mark of the longer sentence is deleted first for a double sentence. If the two sentences are of equal length, the marks at the end of the two sentences are deleted in turn until the total length reaches the requirement (that is, the number of marks deleted from the two sentences of equal length is as equal as possible); if the sequence length is too small, a [PAD] mark is added at the end of the sentence to make the length meet the requirement. Then add a [CLS] mark at the beginning of the sequence, and a [SEP] mark at the end of each sentence. Add one [CLS] and one [SEP] to a single sentence, and one [CLS] and two [SEP] to a double sentence. The representation corresponding to the [CLS] mark is used as the representation of the entire sequence, and the [SEP] mark is specifically used to separate sentences. When processing the length, the added [CLS] and [SEP] tags need to be considered so that the final total length is equal to seq_length. The [PAD] tag is at the end of the entire sequence. For example, the input sentence to be simplified is: "Chechecheche enters bankruptcy liquidation procedure", and after tagging, it is obtained: "[CLS]Chechecheche enters bankruptcy liquidation procedure [SEP]".

[0058] Then, a syntactic embedding is determined based on the labeled first-granularity sentence elements, where the syntactic embedding includes a vector indicating a dependency relationship of each first-granularity sentence element.

[0059] In one embodiment, the generation method of syntactic embedding can refer to Figure 3 Description. First, perform syntactic structure analysis, for example, use the Language Technology Platform (LTP) tool to perform syntactic structure analysis on the input sentence. LTP provides a series of Chinese natural language processing tools, and users can use these tools to perform word segmentation, part-of-speech tagging, syntactic analysis, and other tasks on Chinese text. From an application perspective, LTP provides users with the following components: tools for generating statistical machine learning models for a single natural language processing task; programming interfaces for calling models for analysis for a single natural language processing task; combining various analysis tools in a pipeline manner to form a unified Chinese natural language processing system; system-callable model files for Chinese language processing; and cloud-based programming interfaces for a single natural language processing task. See Figure 3The bottom line shows the results of part-of-speech analysis, while the arcs above and the words on the arcs are the results of syntactic analysis. "Cheche" is the "ATT" of "qiche", where "ATT" indicates the attributive-middle relationship; "qiche" is the "SBV" of "jinru", where "SBV" indicates the subject-predicate relationship; "jinru" is the "HED" of this input, where "HED" indicates the core relationship; "pochan" is the "FOB" of "qingsuan", where "FOB" indicates the preposed object; "qingsuan" is the "ATT" of "chengxu", where "ATT" indicates the attributive-middle relationship; "chengxu" is the "VOB" of "jinru", where "VOB" indicates the verb-object relationship.

[0060] Then, continuing with the above embodiment, according to the parsing results, the dependency relationship of the first-granularity sentence elements (i.e., tokens) is the same as that of the corresponding words. For example, if the dependency relationship of "cheche" is "ATT", then the dependency relationships of the two tokens "che" and "che" are both "ATT". The dependency relationships of [CLS] and [SEP] are recorded as "0". In this way, the correspondence between tokens and dependency relationships in the entire input text "cheche qiche jinru pochan qingsuan chengxu" is as follows:

[0061] {"[CLS]": "O", "che": "ATT", "che": "ATT", "qi": "SBV", "che": "SBV", "jin": "HED", "ru": "HED", "po": "FOB", "chan": "FOB", "qing": "ATT", "suan": "ATT", "cheng": "VOB", "xu": "VOB", "SEP": "0"}.

[0062] In this way, each token has a corresponding dependency relationship, and thus a corresponding vector, from which the syntactic dependency matrix of the entire input sentence is formed, that is, the syntactic embedding in this article, denoted as E 0 , E ATT , E ATT , E SBV , E SBV , E HED , E HED， E FOB , E FOB , E ATT , E ATT , E VOB , E BOV , E 0 (with dimensions of the length after sentence tokenization x the number of hidden layer nodes), this syntactic embedding is a learnable variable, randomly initialized, and then trained together with the downstream task.

[0063] Then, based at least on the syntactic embedding, the topic role of each first-granularity sentence element is predicted via a trained natural language processing model. Here, the topic role indicates the relationship between the first-granularity sentence element and the topic of the sentence to be simplified. For example, the topic role can indicate that a certain first-granularity sentence element is the first first-granularity sentence element of the topic, the middle first-granularity sentence element of the topic, or an off-topic element. In the above example, the topic role predicted by the first "car" is the first first-granularity sentence element of the topic, and the topic role predicted by the second "car", "auto", "car", "broken", "production", "clearance" and "calculation" is the middle first-granularity sentence element of the topic. The other first-granularity sentence elements are off-topic elements.

[0064] In the field of natural language processing, natural language representation models are gaining more and more attention because they can enable machines to better understand human language. Among them, the most representative one is the BERT model. Figure 2 The main model of the natural language processing model illustrated in the figure is the BERT model, however, the embodiments of the present disclosure are not limited to this model, and other models may be used, such as the ERNIE (Enhanced Representation from kNowledge IntEgration) model or the RoBERT (Robustly Optimized BERT Pretraining) model.

[0065] BERT (Bidirectional Encoder Representations from Transformers) is a deep language representation model launched by Google in October 2018. From the original article released by Google: Pre-training of DeepBidirectional Transformers for Language Understanding. BERT is divided into a basic version and a large version. The differences are reflected in the different number of network layers, the size of the hidden layer, and the number of attention heads.

[0066] According to the model released by Google, BERT's input embedding consists of three parts: token embedding, segment embedding, and position embedding, which are obtained by adding these three parts.

[0067] First, the input sequence (i.e., the sentence to be simplified) is tokenized. The tokenization method refers to the method described above. After tokenizing the sentence "Cheche Automobile enters the bankruptcy liquidation process", we get "[CLS]Cheche Automobile enters the bankruptcy liquidation process[SEP]". After the token sequence is input into the BERT model, these tokens are mapped into vectors through one-hot encoding or Word2Vec, resulting in a vector with dimensions seq_length x the number of hidden units of the BERT model, which is called token embedding and denoted as E [cls] , E 车 , E 车 , E 汽 , E 车 , E 进 , E 入， E 破 , E 产 , E 清 , E 算 , E 程 , E 序 , E [sep] . It can be seen that the token embedding includes the word vectors of each of the first-granularity sentence elements.

[0068] BERT also determines the segment embedding of the input token sequence. The segment embedding indicates which segment the token belongs to, represented by natural numbers. If there is only one segment, it is 0; for two segments, it is 0 and 1, and so on. Here, the segment can be a sentence or a part of a sentence. In an example, the segment is a sentence, and the segment embedding is denoted as E A , E A , E A , E A , E A , E A , E A , E A , E A , E A , E A , E A , E A , E A .

[0069] BERT also determines the position embedding of the input sequence. The position embedding indicates the position of this token in the sentence, where the sine and cosine functions used by Transformer are used to represent the vector and are learned. In an example, the position embedding is denoted as E 0 , E 1 , E 2 , E 3 , E 4 , E 5 , E6 , E 7 , E 8 , E 9 , E 10 , E 11 , E 12 , E 13 .

[0070] BERT then adds the above token embedding, fragment embedding, and position embedding to get the sum of the embeddings as the input embedding of BERT, as shown in Figure 4a shown.

[0071] According to an embodiment of the present disclosure, the above-mentioned syntactic embedding, tag embedding, fragment embedding and position embedding are added together to obtain the sum of the embeddings as the input embedding of BERT, such as Figure 4b As shown in the figure, since the input embedding includes syntactic embedding, it is helpful to take syntactic factors into consideration when simplifying sentences, which can help improve the accuracy and readability of sentence simplification.

[0072] The BERT model uses the Transformer encoder as the main model structure. The Transformer model is a brand-new model proposed by Google in the paper "Attention Is All You Need". It abandons the convolutional neural network and recurrent neural network used in previous deep learning tasks, and completely models a piece of text based on the attention mechanism. The core idea of ​​the attention mechanism used by Transformer is to calculate the relationship between each word in a sentence and all the words in the sentence, and then believe that the relationship between these words reflects the correlation and importance between different words in the sentence to a certain extent. Therefore, by using these relationships to adjust the importance (weight) of each word, a new expression for each word can be obtained. This new representation not only contains the word itself, but also contains the relationship between other words and this word. Therefore, it is a more global expression compared to a simple word vector. Transformer obtains the final text expression by continuously overlapping such an attention mechanism layer and a normal nonlinear layer on the input text. The model structure of Transformer is as follows Figure 5 shown.

[0073] The transformer structure consists of an encoder and a decoder. The encoder consists of N = 6 identical layers, where the layers refer to Figure 5 The units on the left are marked with "Nx". Each layer consists of two sub-layers, namely the multi-head self-attention mechanism and the fully connected forward feedback network. Each sub-layer has a residual connection (Add) and normalization (Norm).

[0074] The decoder has a similar structure to the encoder, but with an additional attention sublayer. The decoder output is the probability distribution of the output word corresponding to position i, and the input is the encoder output and the decoder output corresponding to position i-1. The main difference from the encoder is that the newly added attention adds a mask, which ensures that the prediction of the i-th position will not be exposed to future information.

[0075] In addition to the main encoder and decoder, there is also a data preprocessing part. Transformer proposes two position encoding methods, using sine and cosine functions of different frequencies to directly calculate, or learn a position embedding.

[0076] The innovation of BERT is that it uses bidirectional Transformer for natural language models. Experimental results show that the natural language model trained bidirectionally has a deeper understanding of context than the unidirectional natural language model. BERT introduces a technology called Masked Language Model. Before the emergence of this technology, it was impossible to train a bidirectional language model. As mentioned above, the prototype of Transformer includes two independent mechanisms, an encoder responsible for receiving text as input, and a decoder responsible for predicting the results of the task. The goal of BERT is to generate a language model, so only the encoder mechanism is needed. The encoder of Transformer reads the entire text sequence at once, rather than reading it sequentially from left to right or from right to left. This feature enables the model to learn based on both sides of the word, which is equivalent to a bidirectional function.

[0077] After the input embedding input encoder is processed as described above, the topic role of each tag is predicted, where the topic role indicates the relationship between the tag and the topic of the sentence to be simplified. For example, the topic role may include: the first first-granularity sentence element of the topic (for example, represented by "B-KP"), the middle first-granularity sentence element of the topic (for example, represented by "I-KP"), and the out-of-topic element (for example, represented by "0").

[0078] In one example, the output of the encoder can be passed through Softmax to predict the probability of each tag being each topic role; for example, the probability of "car" being the first first-granularity sentence element of the topic is 90%, the probability of being the middle first-granularity sentence element of the topic is 9%, and the probability of being an off-topic element is 1%, and then the topic role with the highest predicted probability is determined as the topic role of the tag. In this way, the prediction result corresponding to "car car car enters bankruptcy liquidation procedure" in the input tag sequence "[CLS] car car car enters bankruptcy liquidation procedure [SEP]" is "B-KPI-KP I-KP I-KP OO I-KP I-KP I-KP I-KP OO". Then, the tag predicted as the expected topic role is formed into a simplified sentence. It is not difficult to understand that the expected topic role includes the first first-granularity sentence element of the topic and the middle first-granularity sentence element of the topic, so the simplified sentence "car car car bankruptcy liquidation" is obtained. The elements in the simplified sentence are completely extracted from the original sentence, and there are no newly generated words, so the readability is good. According to the method disclosed in the present invention, there is no need to manually define the combination rules of the elements extracted from the sentence, but the simplified topic can be automatically output, which is more efficient and convenient to use.

[0079] In another example, the output of the encoder can be passed through a conditional random field (CRF) to obtain a simplified sentence. In another example, the output of the encoder can be passed through a long short-term memory (LSTM) network and a network spliced ​​with a conditional random field to obtain a simplified sentence.

[0080] Both the ERNIE model and the RoBERT model use the Transformer encoder and support input embedding. Figure 4b The obtained input embedding is also applicable to the ERNIE model and the RoBERT model. Of course, the embodiments of the present disclosure are not limited to the BERT model, the ERNIE model and the RoBERT model, but can be applied to various suitable natural language models, especially models that apply input embedding.

[0081] When training the above model, a sequence labeling method can be adopted, and the simplified phrase (denoted as the topic) is used as the role to be predicted (denoted as KP). For example, for the original sentence: "Another new car-making force has fallen, and Cheche Automobile has entered the bankruptcy liquidation process", the simplified topic: "Cheche Automobile bankruptcy liquidation", then the labeling scheme is: "O O O O O O O O O OO B-KP I-KP I-KP I-KP O O I-KP I-KP I-KP I-KP O O", and each word of the sentence is labeled with the corresponding role, where B-KP represents the first word of the topic, I-KP represents the middle word of the topic, and O represents the word outside the topic.

[0082] According to the solution of the embodiment of the present disclosure, the keyword method is abandoned, and it does not rely on the word segmentation dictionary for keyword extraction, and can automatically discover new words or new phrases.

[0083] Figure 6 A flowchart of a sentence simplification method provided by an embodiment of the present invention is illustrated. First, in step 601, the sentence to be simplified is obtained, for example, including "Cheche Automobile has entered the bankruptcy liquidation process". Then, in step 602, the sentence to be simplified is marked to obtain a plurality of first-granularity sentence elements. For the sentence to be simplified being Chinese, the first granularity here can be the smallest granularity of the sentence element - the word. Of course, the embodiment of the present disclosure does not exclude the possibility of other granularities. The marking method can, for example, refer to the marking method involved in the description above about Figure 2 to obtain "[CLS]Cheche Automobile has entered the bankruptcy liquidation process[SEP]". Of course, the embodiment of the present disclosure is not limited thereto, and may also include any other suitable method. In step 603, the syntactic embedding of the sentence to be simplified is determined, and the syntactic embedding includes a vector indicating the dependency relationship of each first-granularity sentence element. The method for determining the syntactic embedding is, for example, the method further described in Figure 7 . Of course, the embodiment of the present disclosure is not limited thereto, and may also include any other suitable method.

[0084] In step 604, at least based on the determined syntactic embedding, the topic role of each first-granularity sentence element is predicted via a trained natural language processing model. As described above, the topic role indicates the relationship between the first-granularity sentence element and the topic of the sentence to be simplified. For example, the topic role can indicate that a certain first-granularity sentence element is the first first-granularity sentence element of the topic, the middle first-granularity sentence element of the topic, or an off-topic element. In the above example, the topic role predicted by the first "car" is the first first-granularity sentence element of the topic, and the topic role predicted by the second "car", "car", "car", "bankruptcy", "production", "clearance" and "calculation" is the middle first-granularity sentence element of the topic. The other first-granularity sentence elements are off-topic elements. In step 605, the first-granularity sentence elements predicted as the expected topic role are used to form a simplified sentence. For example, according to the above prediction results, the simplified sentence obtained is: "car car car bankruptcy liquidation".

[0085] Figure 7 The flowchart of determining syntactic embedding provided by an embodiment of the present invention is shown. In step 6031, based on the second granularity, the sentence to be simplified is subjected to dependency syntactic analysis to obtain dependency relationships of the sentence elements of the second granularity. The method for confirming the dependency relationship can refer to the above description of Figure 3 Described manner. Of course, the embodiments of the present disclosure are not limited to this, and may also include any other suitable manner. In one example, the second granularity is greater than the first granularity, for example, the first granularity is a character, and the second granularity is a word, then each second granularity sentence element includes one or more first granularity sentence elements. In step 6032, the dependency relationship of each second granularity sentence element is determined as the dependency relationship of each first granularity sentence element included in the second granularity sentence element, and a corresponding vector of each first granularity sentence element is generated based on the dependency relationship, thereby obtaining a syntactic embedding. Such a syntactic embedding includes a vector indicating the dependency relationship of each of the first granularity sentence elements.

[0086] Figure 8 FIG. 6 is another flowchart of the sentence simplification method provided by an embodiment of the present invention. Figure 6 The same as described in , will not be repeated here. In step 801, determine the token embedding of the sentence to be simplified, the token embedding includes the word vector of each first granularity sentence element. The method of determining the token embedding can refer to the above about Figure 2 The description of the simplified sentence is omitted here. Of course, the embodiments of the present disclosure are not limited to this, and may also include any other suitable methods. In step 802, the fragment embedding of the sentence to be simplified is determined. The fragment embedding includes a vector indicating which fragment each first-granularity sentence element belongs to. The fragment includes a sentence or a component of a sentence. The method for determining the fragment embedding can refer to the above description of Figure 2 description is not repeated here. Of course, the embodiments of the present disclosure are not limited thereto, and may also include any other suitable manner. In step 803, the positional embedding of the sentence to be simplified is determined. The positional embedding includes vectors indicating the positions of each first-granularity sentence element. The determination method of the positional embedding can refer to the foregoing description of Figure 2 description is not repeated here. Of course, the embodiments of the present disclosure are not limited thereto, and may also include any other suitable manner. In step 804, the syntactic embedding obtained in step 603, the token embedding obtained in step 801, the segment embedding obtained in step 802, and the positional embedding obtained in step 804 are added together to obtain the sum of the embeddings. Then, in step 805, based on the sum of the embeddings, via a trained natural language processing model, the topic role of each first-granularity sentence element is predicted. For example, the topic role predicted for the first "car" is the first first-granularity sentence element of the topic, and the topic roles predicted for the second "car", "steam", "car", "bankruptcy", "liquidation" are the middle first-granularity sentence elements of the topic. And other first-granularity sentence elements are out-of-topic elements. Here, the prediction method and the trained natural language processing model can refer to the foregoing description of Figure 2 description is not repeated here. Of course, the embodiments of the present disclosure are not limited thereto, and may also include any other suitable manner. Finally, in step 605, the first-granularity sentence elements predicted to be the desired topic roles are used to form the simplified sentence. For example, according to the above prediction results, the simplified sentence obtained is: "Car car automobile bankruptcy liquidation".

[0087] Fig. 9 The figure illustrates a flowchart for predicting topic roles provided by an embodiment of the present invention. In step 6041, the probability of each first-granularity sentence element as each topic role is predicted. For example, the probability that "car" is predicted as the first first-granularity sentence element of the topic is 90%, the probability as the middle first-granularity sentence element of the topic is 9%, and the probability as an out-of-topic element is 1%. In step 6042, the topic role with the highest predicted probability of each first-granularity sentence element is determined as the topic role of the first-granularity sentence element. For example, in the above example, the probability of the first first-granularity sentence element with the highest predicted probability is determined as the topic role of "car".

[0088] Fig.10The block diagram of the sentence simplification device provided by an embodiment of the present invention is illustrated. The sentence simplification device 1000 includes a sentence acquisition module 1001, a labeling module 1002, a syntactic embedding determination module 1003, a prediction module 1004 and a formation module 1005. The sentence acquisition module 1001 is configured to acquire a sentence to be simplified, the labeling module 1002 is configured to label the sentence to be simplified to obtain a plurality of first-granularity sentence elements, the syntactic embedding determination module 1003 is configured to determine the syntactic embedding of the sentence to be simplified, and the syntactic embedding includes a vector indicating the dependency relationship of each first-granularity sentence element. The prediction module 1004 is configured to predict the topic role of each first-granularity sentence element based on at least the syntactic embedding via a trained natural language processing model, and the topic role indicates the relationship between the first-granularity sentence element and the topic of the sentence to be simplified. The formation module 1005 is configured to form a simplified sentence using the first-granularity sentence element predicted to be the desired topic role.

[0089] Fig.11 FIG. 1 is a schematic diagram of a hardware implementation environment according to an embodiment of the present invention. Fig.11 In an embodiment of the present invention, the sentence simplification device 1102 includes a processor 1104, which includes a hardware element 1110. The processor 1104 includes, for example, one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. The term "processor" as used herein may refer to any of the above structures or any other structures suitable for implementing the technology described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated hardware and / or software module configured for sentence simplification, or incorporated in a combined hardware and / or software module. Moreover, the technology may be fully implemented in one or more circuits or logic elements. The method in the present disclosure may be implemented in various components, modules, or units, but it does not necessarily need to be implemented by different hardware units. Instead, as described above, various components, modules, or units may be combined or provided by a collection of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware.

[0090] In one or more examples, the above combination Figure 6-Figure 9The described technical solutions can be implemented in hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium 1106 and executed by a hardware-based processor. The computer-readable medium 1106 may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium containing any medium that facilitates the transmission of a computer program from one place to another, such as according to a communication protocol. In this way, the computer-readable medium 1106 can generally correspond to (1) a non-temporary tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. The data storage medium can be any available medium that can be read by one or more computers or one or more processors to retrieve instructions, codes and / or data structures for implementing the technology described in the present disclosure. A computer program product may include a computer-readable medium 1106.

[0091] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disks, magnetic disk storage or other magnetic storage, flash memory, or any other memory 1112 that can be used to store desired program code in the form of instructions or data structures and that can be read by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium 1106. For example, if instructions are transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other transient media, but are directed to non-transient tangible storage media. As used herein, disks and optical disks include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks, where disks typically reproduce data magnetically, while optical disks use lasers to reproduce data optically. Combinations of the above should also be included within the scope of computer-readable media 1106 .

[0092] The statement simplification device 1102 may also be provided in the statement simplification device 1100 together with an I / O interface 1106 for transmitting data, and other functions 1114. The statement simplification device 1100 may be included in different terminals, such as a desktop, a mobile terminal, etc., where a desktop 1116, a mobile terminal 1118, and other terminals 1120 are illustrated. Each of these configurations includes devices that may have generally different structures and capabilities, and thus the statement simplification device 1100 may be configured according to one or more of the different terminals. In addition, the technology of the present invention may also be implemented in whole or in part on a "cloud" 1122 by using a distributed system, such as through a platform 1124 as described below.

[0093] The cloud 1122 includes and / or represents a platform 1124 for resources 1126. The platform 1124 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of the cloud 1122. The resources 1126 may include applications and / or data that may be used when executing computer processing on a server remote from the computing device 1102. The resources 1126 may also include services provided over the Internet and / or over a subscriber network such as a cellular or Wi-Fi network.

[0094] The platform 1124 can abstract resources and functions to connect the computing device 1102 with other computing devices. The platform 1124 can also be used to abstract the hierarchy of resources to provide a hierarchy of corresponding levels of demand encountered for resources 1126 implemented via the platform 1124. Therefore, in an interconnected device embodiment, the implementation of the functions described herein can be distributed throughout the system. For example, the functions can be implemented partially on the computing device 1102 and through the platform 1124 that abstracts the functions of the cloud 1122.

[0095] According to the embodiments of the present disclosure, the elements in the simplified sentence are completely extracted from the original sentence, there are no newly generated words, and the readability is good. Since the input embedding includes syntactic embedding, it is beneficial to take syntactic factors into consideration when simplifying the sentence, which can help improve the accuracy and readability of the sentence simplification. According to the method of the present disclosure, there is no need to manually define the combination rules of the elements extracted from the sentence, but the simplified topic can be automatically output, which is more efficient and more convenient to use. According to the scheme of the embodiments of the present disclosure, the keyword method is abandoned, and it does not rely on the word segmentation dictionary used for keyword extraction, and new words or new phrases can be automatically discovered.

[0096] It should be noted that the expressions "first", "second" and the like in the present disclosure do not indicate the importance or the order of steps, but are only used to distinguish. In the absence of special instructions or premise constraints (i.e., the execution of one step requires the execution result of another step as a premise), the order of description of the method steps does not represent their execution order, and the described method steps can be executed in any possible and reasonable order.

[0097] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0098] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A sentence simplification method, characterized in that: Includes steps: Get the statement to be simplified; Marking the sentence to be simplified to obtain a plurality of first-granularity sentence elements, wherein the sentence is a Chinese sentence and the first-granularity sentence elements are characters; Determining the syntactic embedding of the sentence to be simplified, comprising: performing dependency syntactic analysis on the sentence to be simplified based on the second granularity to obtain dependency relations of sentence elements of the second granularity, wherein each sentence element of the second granularity includes one or more sentence elements of the first granularity, determining the dependency relations of each sentence element of the second granularity as the dependency relations of each sentence element of the first granularity included in the sentence element of the second granularity, and generating a corresponding vector of each sentence element of the first granularity according to the dependency relations, thereby obtaining the syntactic embedding; Based at least on the syntactic embedding, predicting a topic role of each first-granularity sentence element, the topic role indicating a relationship between the first-granularity sentence element and a topic of the sentence to be simplified, and including: a first first-granularity sentence element of a topic, an intermediate first-granularity sentence element of a topic, and an off-topic element; and A simplified sentence is formed using first granularity sentence elements predicted to be the desired topic role.

2. The method according to claim 1, characterized in that The method further comprises: Determining a token embedding of the sentence to be simplified, wherein the token embedding includes a word vector of each sentence element of the first granularity; Determining a fragment embedding of the statement to be simplified, the fragment embedding comprising a vector indicating to which fragment each of the first granularity statement elements belongs, the fragment comprising a statement or a component of a statement; Determine a position embedding of the sentence to be simplified, wherein the position embedding includes a vector indicating a position of each sentence element of the first granularity; The prediction further includes: A topic role of each first-granularity sentence element is predicted based on at least the syntactic embedding, the token embedding, the fragment embedding, and the position embedding.

3. The method according to claim 2, characterized in that The forecast further includes: adding the syntactic embedding, the token embedding, the fragment embedding and the position embedding to obtain a sum of embeddings; and Based on the sum of the embeddings, the topic role of each first-granularity sentence element is predicted.

4. The method according to claim 1, characterized in that The second granularity sentence element is a word.

5. The method according to claim 1, characterized in that The forecast further includes: The syntactic embedding is used as at least a part of the input embedding of a trained natural language processing model, and the topic role of each first-granularity sentence element is predicted via the natural language processing model, wherein the natural language processing model includes any one of the following: a BERT model, an ERNIE model, or a RoBERT model.

6. The method according to claim 5, characterized in that The forecast further includes: Predicting the probability of each of the first granularity sentence elements being a role of each topic; and The topic role with the highest predicted probability for each of the first-granularity sentence elements is determined as the topic role of the first-granularity sentence element.

7. The method according to claim 1, characterized in that The expected topic role includes: a first first-granularity sentence element of the topic and an intermediate first-granularity sentence element of the topic.

8. A sentence simplification device, characterized in that: include: A statement acquisition module, configured to acquire statements to be simplified; a marking module configured to mark the sentence to be simplified to obtain a plurality of first-granularity sentence elements, wherein the sentence is a Chinese sentence and the first-granularity sentence elements are characters; A syntactic embedding determination module is configured to determine the syntactic embedding of the sentence to be simplified, the determination comprising: based on the second granularity, performing dependency syntactic analysis on the sentence to be simplified to obtain dependency relations of second granularity sentence elements, wherein each of the second granularity sentence elements includes one or more first granularity sentence elements, determining the dependency relations of each of the second granularity sentence elements as the dependency relations of each of the first granularity sentence elements included in the second granularity sentence element, and generating a corresponding vector of each of the first granularity sentence elements according to the dependency relations, thereby obtaining the syntactic embedding; a prediction module configured to predict a topic role of each first-granularity sentence element based at least on the syntactic embedding, the topic role indicating a relationship between the first-granularity sentence element and a topic of the sentence to be simplified, and comprising: a first first-granularity sentence element of a topic, an intermediate first-granularity sentence element of a topic, and an off-topic element; and A forming module is configured to form a simplified sentence using first-granularity sentence elements predicted to be of a desired topic role.

9. A device for sentence simplification, characterized in that: include: processor; as well as A memory configured to store thereon computer executable instructions, which, when executed in the processor, enable the method according to any one of claims 1 to 7 to be implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the computer-readable storage medium is executed on a computer, the computer is enabled to implement the method according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images