Cross-language automatic summary generation model establishment method and summary generation method
By preprocessing and similarity judgment on the cross-language document-abstract parallel dataset and generating and updating the pseudo-reference summary, the problem of insufficient error propagation and bilingual information utilization in the existing cross-language automatic summary method is solved, and a higher quality and accurate cross-language automatic summary is achieved.
Patent Information
- Application Number
- CN202011309483.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-10-30
- Filing Date
- 2020-11-20
- Publication Date
- 2025-03-21
- Estimated Expiration
- 2040-11-20
AI Technical Summary
Existing cross-language automatic digest methods are prone to error propagation in the absence of parallel data, and multitasking learning-based methods fail to make full use of bilingual information.
By preprocessing the cross-language document-abstract parallel dataset, a pseudo-reference summary of the source and target languages is generated, and the pseudo-reference summary is retained or discarded according to the similarity judgment, the dataset is updated, and finally trained in the encoder & decoder framework to establish a cross-language automatic summary generation model.
It effectively solves the error propagation problem, makes full use of bilingual information, and improves the quality and accuracy of cross-language automatic summary.
Smart Images
Figure CN112364158B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of natural language processing, and relates to a method for establishing a cross-language automatic summary generation model, a cross-language automatic summary generation method, a device, an electronic device and a storage medium. Background Art
[0002] Cross-lingual automatic summarization is a task to extract the core information of the source language document and organize it into a summary in the target language (different from the source language).
[0003] Due to the lack of parallel data, most cross-language automatic summarization methods can only be implemented using a pipeline approach, which easily leads to error propagation problems.
[0004] In recent years, with the construction of large-scale cross-language automatic summarization datasets, some studies have begun to focus on cross-language automatic summarization methods based on deep learning. One of the more typical methods is based on multi-task learning, which jointly optimizes the cross-language automatic summarization model with the monolingual automatic summarization model and achieves good performance. However, this method uses two independent decoders to generate summaries in the source language and the target language for the source document at the same time, which makes it impossible to fully utilize the bilingual information. Summary of the invention
[0005] In order to solve at least one of the above technical problems, the present disclosure provides a method for establishing a cross-language automatic summary generation model, a cross-language automatic summary generation method, a device, an electronic device and a storage medium.
[0006] According to one aspect of the present disclosure, a method for establishing a cross-language automatic summary generation model is provided, comprising: S1, preprocessing document-summary data in a cross-language document-summary parallel data set, wherein the cross-language document-summary parallel data set comprises a plurality of source language document-target language reference summary parallel data, wherein the source language document-target language reference summary parallel data comprises a source language document and a target language reference summary corresponding to the source language document; S2, processing each of the source language document-target language reference summary parallel data: machine translating the target language reference summary into a source language pseudo reference summary, and then machine translating the source language pseudo reference summary into a target language pseudo reference summary; S3 , determine the similarity between the target language pseudo-reference summary of each source language document-target language reference summary parallel data and the target language reference summary of the source language document-target language reference summary parallel data, and based on the similarity, retain or discard the source language pseudo-reference summary of the source language document-target language reference summary parallel data, and update the cross-language document-summary parallel data set; and S4, at least input all [source language document-target language reference summary-source language pseudo-reference summary] triplets in the updated cross-language document-summary parallel data set into the encoder & decoder framework for training, and obtain the cross-language automatic summary generation model after the training converges.
[0007] According to at least one embodiment of the cross-language automatic summary generation model establishment method of the present disclosure, based on the similarity, the source language pseudo-reference summary of the source language document-target language reference summary parallel data is retained or discarded, including: when the similarity is lower than a preset threshold, the source language pseudo-reference summary of the source language document-target language reference summary parallel data is discarded; when the similarity is greater than or equal to the preset threshold, the source language pseudo-reference summary of the source language document-target language reference summary parallel data is retained.
[0008] According to at least one embodiment of the cross-language automatic summary generation model establishment method of the present disclosure, based on the similarity, the source language pseudo-reference summary of the source language document-target language reference summary parallel data is retained or discarded, including: when the similarity is lower than or equal to a preset threshold, the source language pseudo-reference summary of the source language document-target language reference summary parallel data is discarded; when the similarity is greater than the preset threshold, the source language pseudo-reference summary of the source language document-target language reference summary parallel data is retained.
[0009] According to the method for establishing a cross-language automatic summary generation model of at least one embodiment of the present disclosure, the similarity is a ROUGE indicator.
[0010] According to the method for establishing a cross-language automatic summary generation model of at least one embodiment of the present disclosure, the encoder & decoder framework includes a source language document encoder, a source language summary decoder, and a target language summary decoder, and different languages use independent vocabularies.
[0011] According to the method for establishing a cross-language automatic summary generation model of at least one embodiment of the present disclosure, the encoder & decoder framework adopts the Transformer architecture.
[0012] According to the method for establishing a cross-language automatic summary generation model of at least one embodiment of the present disclosure, in step S4, at least all the [source language document-target language reference summary-source language pseudo reference summary] triples in the updated cross-language document-summary parallel data set are input into the encoder & decoder framework for training, including: for each source language document in the [source language document-target language reference summary-source language pseudo reference summary] triple, the source language-side summary decoder and the target language-side summary decoder synchronously perform summary decoding, and in the decoding process, the target language-side summary decoder fuses the source language document, the target language summary vocabulary generated by the target language-side summary decoder, and the source language summary vocabulary generated by the source language-side summary decoder at the decoding time to predict the target language summary vocabulary at the decoding time, and the source language-side summary decoder fuses the source language document, the target language summary vocabulary generated by the target language-side summary decoder, and the source language summary vocabulary generated by the source language-side summary decoder at the decoding time to predict the source language summary vocabulary at the decoding time.
[0013] According to a method for establishing a cross-language automatic summary generation model in at least one embodiment of the present disclosure, the target language-side summary decoder fuses the source language document, the target language summary vocabulary generated by the target language-side summary decoder, and the source language summary vocabulary generated by the source language-side summary decoder at the decoding moment to predict the target language summary vocabulary at the decoding moment, including: the target language-side summary decoder calculates the attention distribution of the generated source language summary vocabulary and calculates the semantic vector of the generated source language summary vocabulary, and the target language-side summary decoder performs the fusion based at least on the attention distribution and the semantic vector of the generated source language summary vocabulary.
[0014] According to a method for establishing a cross-language automatic summary generation model of at least one embodiment of the present disclosure, the source language-side summary decoder fuses the source language document, the target language summary vocabulary generated by the target language-side summary decoder, and the source language summary vocabulary generated by the source language-side summary decoder at the decoding moment to predict the source language summary vocabulary at the decoding moment, including: the source language-side summary decoder calculates the attention distribution of the generated target language summary vocabulary and calculates the semantic vector of the generated target language summary vocabulary, and the source language-side summary decoder performs the fusion based on at least the attention distribution and the semantic vector of the generated target language summary vocabulary.
[0015] According to the method for establishing a cross-language automatic summary generation model of at least one embodiment of the present disclosure, the source language summary and the target language summary predicted by the cross-language automatic summary generation model calculate the negative log-likelihood loss based on the source language pseudo-reference summary and the target language reference summary, and based on the negative log-likelihood loss, the parameters of the model are updated using a back-propagation algorithm.
[0016] According to another aspect of the present disclosure, a cross-language automatic summary generation method is provided, which uses a cross-language automatic summary generation model established by any of the cross-language automatic summary generation model establishment methods described above to synchronously decode an input source language document to generate a source language summary and a target language summary.
[0017] According to another aspect of the present disclosure, a device for establishing a cross-language automatic summary generation model is provided, comprising: a preprocessing module, the preprocessing module preprocesses document-summary data in a cross-language document-summary parallel data set, the cross-language document-summary parallel data set includes a plurality of source language document-target language reference summary parallel data, the source language document-target language reference summary parallel data includes a source language document and a target language reference summary corresponding to the source language document; a translation module, the translation module processes each of the source language document-target language reference summary parallel data: machine translates the target language reference summary into a source language pseudo reference summary, and then machine translates the source language pseudo reference summary into a target language pseudo reference summary; a similarity judgment module The cross-language document-summary parallel dataset comprises a block, the similarity judgment module judges the similarity between the target language pseudo reference summary of each source language document-target language reference summary parallel data and the target language reference summary of the source language document-target language reference summary parallel data, and based on the similarity, retains or discards the source language pseudo reference summary of the source language document-target language reference summary parallel data, and updates the cross-language document-summary parallel dataset; and an encoder & decoder framework module, at least inputs all the [source language document-target language reference summary-source language pseudo reference summary] triples in the updated cross-language document-summary parallel dataset into the encoder & decoder framework module for training, and obtains the cross-language automatic summary generation model after the training converges.
[0018] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes any one of the methods described above.
[0019] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and the execution instructions are used to implement any of the above-mentioned methods when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.
[0021] Figure 1 The present invention is a flowchart of a method for establishing a cross-language automatic summary generation model according to an embodiment of the present invention.
[0022] Figure 2 The present invention is a flowchart of a method for cross-language automatic summary generation according to an embodiment of the present invention.
[0023] Figure 3 The figure is a schematic diagram of bilingual information interaction according to an example of the present disclosure.
[0024] Figure 4 A schematic diagram of the structure of a device for establishing a cross-language automatic summary generation model according to an embodiment of the present disclosure.
[0025] Figure 5 An example diagram of a cross-language automatic summary generation apparatus using a hardware implementation of a processing system is shown.
[0026] Description of Reference Numerals
[0027] 1000 Cross-language automatic summary generation device
[0028] 1001 Preprocessing Module
[0029] 1002 Translation Module
[0030] 1003 Similarity judgment module
[0031] 1004 Encoder & Decoder Framework Module
[0032] 1100 Bus
[0033] 1200 processor
[0034] 1300 Memory
[0035] 1400 Other circuits. DETAILED DESCRIPTION
[0036] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.
[0037] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0038] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present disclosure.
[0039] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.
[0040] When a component is referred to as being "on," "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.
[0041] The terms used herein are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values and / or the values provided that will be recognized by those of ordinary skill in the art.
[0042] Figure 1 It is a flowchart of a method for establishing a cross-language automatic summary generation model according to an embodiment of the present disclosure.
[0043] like Figure 1 As shown, the cross-language automatic summary generation model establishment method 100 includes:
[0044] S1. Preprocessing the document-summary data in the cross-language document-summary parallel dataset, where the cross-language document-summary parallel dataset includes a plurality of source language document-target language reference summary parallel data, where the source language document-target language reference summary parallel data includes a source language document and a target language reference summary corresponding to the source language document;
[0045] S2, processing each source language document-target language reference summary parallel data: machine translating the target language reference summary into a source language pseudo reference summary, and then machine translating the source language pseudo reference summary into a target language pseudo reference summary;
[0046] S3, determining the similarity between the target language pseudo reference summary of each source language document-target language reference summary parallel data and the target language reference summary of the source language document-target language reference summary parallel data, and based on the similarity, retaining or discarding the source language pseudo reference summary of the source language document-target language reference summary parallel data, and updating the cross-language document-summary parallel data set; and
[0047] S4. Input at least all the [source language document-target language reference summary-source language pseudo reference summary] triplets in the updated cross-language document-summary parallel dataset into the encoder & decoder framework for training. After the training converges, a cross-language automatic summary generation model is obtained.
[0048] The preprocessing of the document-summary data in the cross-language document-summary parallel dataset may include performing word segmentation processing and / or lowercase processing on the document-summary data.
[0049] According to a preferred embodiment of the present disclosure, the encoder & decoder framework includes a source language-side document encoder, a source language-side summary decoder, and a target language-side summary decoder, and different languages use independent vocabularies.
[0050] The source-side vocabulary can be obtained by counting the high-frequency words in the source language document. The target-side vocabulary can be obtained by counting the high-frequency words in the target language document (the target language document can be obtained by machine translation of the source language document). The word frequency can be preset.
[0051] The source language may be English, for example, and the target language may be Chinese, for example.
[0052] According to a preferred embodiment of the present disclosure, the similarity is the ROUGE indicator. Those skilled in the art should understand that the ROUGE indicator (Recall-Oriented Understudy for Gisting Evaluation) is mainly based on the recall rate.
[0053] According to a preferred embodiment of the present disclosure, the encoder & decoder framework adopts the Transformer architecture.
[0054] According to a preferred embodiment of the present disclosure, a method 100 for establishing a cross-language automatic summary generation model includes:
[0055] S1. Preprocessing the document-summary data in the cross-language document-summary parallel dataset, where the cross-language document-summary parallel dataset includes a plurality of source language document-target language reference summary parallel data, where the source language document-target language reference summary parallel data includes a source language document and a target language reference summary corresponding to the source language document;
[0056] S2, processing each source language document-target language reference summary parallel data: machine translating the target language reference summary into a source language pseudo reference summary, and then machine translating the source language pseudo reference summary into a target language pseudo reference summary;
[0057] S3, determining the similarity between the target language pseudo reference summary of each source language document-target language reference summary parallel data and the target language reference summary of the source language document-target language reference summary parallel data, and based on the similarity, retaining or discarding the source language pseudo reference summary of the source language document-target language reference summary parallel data, and updating the cross-language document-summary parallel data set; and
[0058] S4. Input at least all the [source language document-target language reference summary-source language pseudo reference summary] triplets in the updated cross-language document-summary parallel dataset into the encoder & decoder framework for training. After the training converges, a cross-language automatic summary generation model is obtained.
[0059] Wherein, in step S3, based on the similarity, retaining or discarding the source language pseudo reference summary of the source language document-target language reference summary parallel data includes:
[0060] When the similarity is lower than a preset threshold, the source language pseudo-reference summary of the source language document-target language reference summary parallel data is discarded; when the similarity is greater than or equal to the preset threshold, the source language pseudo-reference summary of the source language document-target language reference summary parallel data is retained.
[0061] According to a preferred embodiment of the present disclosure, a method 100 for establishing a cross-language automatic summary generation model includes:
[0062] S1. Preprocessing the document-summary data in the cross-language document-summary parallel dataset, where the cross-language document-summary parallel dataset includes a plurality of source language document-target language reference summary parallel data, where the source language document-target language reference summary parallel data includes a source language document and a target language reference summary corresponding to the source language document;
[0063] S2, processing each source language document-target language reference summary parallel data: machine translating the target language reference summary into a source language pseudo reference summary, and then machine translating the source language pseudo reference summary into a target language pseudo reference summary;
[0064] S3, determining the similarity between the target language pseudo reference summary of each source language document-target language reference summary parallel data and the target language reference summary of the source language document-target language reference summary parallel data, and based on the similarity, retaining or discarding the source language pseudo reference summary of the source language document-target language reference summary parallel data, and updating the cross-language document-summary parallel data set; and
[0065] S4. Input at least all the [source language document-target language reference summary-source language pseudo reference summary] triplets in the updated cross-language document-summary parallel dataset into the encoder & decoder framework for training. After the training converges, a cross-language automatic summary generation model is obtained.
[0066] Wherein, in step S3, based on the similarity, retaining or discarding the source language pseudo reference summary of the source language document-target language reference summary parallel data includes:
[0067] When the similarity is lower than or equal to a preset threshold, the source language pseudo-reference summary of the source language document-target language reference summary parallel data is discarded; when the similarity is greater than the preset threshold, the source language pseudo-reference summary of the source language document-target language reference summary parallel data is retained.
[0068] In each of the above implementations, preferably, in step S4, at least all the [source language document-target language reference summary-source language pseudo reference summary] triplets in the updated cross-language document-summary parallel dataset are input into the encoder & decoder framework for training, including:
[0069] For each source language document in the [source language document-target language reference summary-source language pseudo-reference summary] triple, the source language side summary decoder and the target language side summary decoder synchronously perform summary decoding, and in the decoding process, the target language side summary decoder fuses the source language document, the target language summary vocabulary generated by the target language side summary decoder, and the source language summary vocabulary generated by the source language side summary decoder at the decoding moment to predict the target language summary vocabulary at the decoding moment, and the source language side summary decoder fuses the source language document, the target language summary vocabulary generated by the target language side summary decoder, and the source language summary vocabulary generated by the source language side summary decoder at the decoding moment to predict the source language summary vocabulary at the decoding moment.
[0070] In the above implementation manner, preferably, the target language end summary decoder fuses the source language document, the target language summary vocabulary generated by the target language end summary decoder, and the source language summary vocabulary generated by the source language end summary decoder at the decoding moment to predict the target language summary vocabulary at the decoding moment, including:
[0071] The target language end summary decoder performs attention distribution calculation on the generated source language summary vocabulary, and calculates the semantic vector of the generated source language summary vocabulary, and the target language end summary decoder performs fusion based on at least the attention distribution and semantic vector of the generated source language summary vocabulary. In the above implementation, preferably, the source language end summary decoder fuses the source language document, the target language summary vocabulary generated by the target language end summary decoder, and the source language summary vocabulary generated by the source language end summary decoder at the decoding moment to predict the source language summary vocabulary at the decoding moment, including:
[0072] The source language side summary decoder calculates the attention distribution of the generated target language summary vocabulary and calculates the semantic vector of the generated target language summary vocabulary. The source language side summary decoder performs fusion based on at least the attention distribution and semantic vector of the generated target language summary vocabulary.
[0073] More preferably, the target language side summary decoder performs attention distribution calculation on the generated source language summary vocabulary and calculates the semantic vector of the generated source language summary vocabulary, and the source language side summary decoder performs attention distribution calculation on the generated target language summary vocabulary and calculates the semantic vector of the generated target language summary vocabulary.
[0074] The target language summary decoder predicts the target language summary vocabulary at the decoding moment based on the semantic weights of the semantic vectors of the generated source language summary vocabulary, the semantic weights of the semantic vectors of the generated target language summary vocabulary, and the semantics of the source language document (mapping to the target language vocabulary table through a softmax function).
[0075] The source language summary decoder predicts the source language summary vocabulary at the decoding moment based on the semantic weights of the semantic vectors of the generated source language summary vocabulary, the semantic weights of the semantic vectors of the generated target language summary vocabulary, and the semantics of the source language document (mapped to the source language vocabulary through a softmax function). In the above implementation, the source language vocabulary can be obtained by counting the high-frequency words of the source language document. The target language vocabulary can be obtained by counting the high-frequency words of the target language document (the target language document can be obtained by machine translation of the source language document). The word frequency can be preset.
[0076] In each of the above-mentioned implementation modes, preferably, the source language summary and the target language summary predicted by the cross-language automatic summary generation model calculate the negative log-likelihood loss according to the source language pseudo-reference summary and the target language reference summary, and based on the negative log-likelihood loss, the parameters of the model are updated using the back-propagation algorithm.
[0077] Figure 2 It is a flowchart of a cross-language automatic summary generation method according to an embodiment of the present disclosure.
[0078] like Figure 2 As shown, the cross-language automatic summary generation method 200 uses the cross-language automatic summary generation model established by the cross-language automatic summary generation model establishment method of any of the above embodiments to synchronously decode the input source language document to generate a source language summary and a target language summary.
[0079] The cross-language automatic summary generation method 200 comprises the following steps:
[0080] SS1. Preprocess the input source language document;
[0081] SS2, using a cross-language automatic summarization model to synchronously decode the pre-processed source language document; and
[0082] SS3, generate source language summary and target language summary.
[0083] The source language may be Chinese / English, etc., and the target language may be English / Chinese, etc. The preprocessing may be word segmentation processing and / or lowercase processing.
[0084] The cross-language automatic summary generation model and the cross-language automatic summary generation method of the present disclosure are described in detail below through a specific example.
[0085] The source language is, for example, English, and the target language is, for example, Chinese.
[0086] Take the data “Rod Gray, 94, had been taken to hospital by ambulance after he cut his head in a fall at his home (input document) –> Rod Gray was taken to Ipswich Hospital after falling at home (target language reference abstract)” as an example.
[0087] Firstly, the document-summary data in the cross-language document-summary parallel dataset are preprocessed, including word segmentation and / or lowercase processing.
[0088] Among them, Chinese word segmentation can use Urheen from the National Key Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, and English word segmentation can use the tokenize method in nltk.
[0089] The function of lowercase processing is to convert the uppercase letters contained in the text into lowercase. Through the above two operations, "Rod gray,94,had been taken to hospital by ambulance after he cut hishead in a fall at his home" can be converted to "rod gray,94,had been taken to hospital by ambulance after he cut his head in a fall at his home", and "Rod Gray fell at home and was taken to Ipswich Hospital" can be converted to "Rod Gray fell at home and was taken to Ipswich Hospital".
[0090] The reference abstracts (target language) in the cross-language document-abstract parallel dataset are subjected to a round-trip translation step, that is, they are translated into the source language and then translated back to the target language.
[0091] Among them, the round-trip translation step of the reference abstract (target language) can be performed using a machine translation module in the prior art. For example, first translate "Rod Grey fell at home and was taken to Ipswich Hospital" into "Rod greyfell at home and was taken to Ipswich hospital", and then translate it back to Chinese "Rod Grey fell at home and was taken to Ipswich Hospital" (pseudo reference abstract).
[0092] If the ROUGE value of the new target language pseudo-reference summary obtained in the above step and the original reference summary is higher than the preset threshold, the corresponding source language pseudo-reference summary in the above step is retained, otherwise it is filtered out.
[0093] Among them, the ROUGE value of the new target language pseudo-reference summary "Rod Gray fell at home and was sent to Ipswich Hospital" and the original reference summary "Rod Gray fell at home and was sent to Ipswich Hospital" can be calculated based on Chinese characters. In this example, the similarity between the two sentences is very high, so the pseudo-reference summary is retained.
[0094] During the training phase, all [source language input document - target language reference summary - source language pseudo reference summary] triplets are input into the encoder & decoder framework for model training. The framework is preferably composed of a source language document encoder, a source language summary decoder and a target language summary decoder, and different languages use independent vocabularies.
[0095] The encoder & decoder framework can adopt the Transformer architecture.
[0096] During the training and testing process, the decoders of the two languages perform summary decoding synchronously. At the decoding moment, each decoder will fuse the source information, the information of the vocabulary generated by the decoder, and the information of the vocabulary generated by the other decoder to make vocabulary predictions at the current moment.
[0097] The statement "the decoders of both languages perform abstract decoding synchronously" indicates that when the Chinese abstract decoder generates the word "罗德", the English abstract decoder also generates the word "rod" simultaneously. And the statement "each decoder fuses the source information, the information of the words already generated by this decoder, and the information of the words already generated by the other decoder at the decoding moment" indicates that taking the seventh moment as an example, the Chinese abstract decoder has generated the words "罗德格雷在家中摔倒", while the English abstract has generated "rodgrey fell at home and". Then at the current moment, the Chinese abstract decoder calculates the attention distribution for the words already generated in the English abstract and calculates the semantic vector at the English abstract end, so as to be used to predict the current Chinese word "后"; the prediction for the English abstract words is the same. At the same time, the output predicted by the model will calculate the negative log-likelihood loss based on the source language reference abstract and the target language reference abstract, and then update the model parameters using the backpropagation algorithm based on this loss.
[0098] After the training converges, stop the training process and save the parameters of the model.
[0099] In the test phase, still first preprocess the input document, and use the saved model to perform synchronous decoding on the document. The obtained cross-language abstract is the output.
[0100] Among them, decoding preferably uses the Beam Search algorithm to greedily find an abstract sequence that can maximize the output probability as the final output.
[0101] Figure 3 The schematic diagram of bilingual information interaction of the above example is shown.
[0102] In the process of cross-language abstract generation of the present disclosure, not only the abstract in the target language form is generated, but also the abstract in the source language form is generated, and the complementary effect is realized by using the interaction of bilingual information in the process of generating the abstracts of these two languages, so as to improve the quality of the final target language abstract.
[0103] Figure 4 It is the structural schematic diagram of a cross-language automatic abstract generation model establishment device according to an embodiment of the present disclosure.
[0104] The cross-language automatic abstract generation model establishment device 1000 includes:
[0105] A preprocessing module 1001, and the preprocessing module 1001 preprocesses the document-abstract data in the cross-language document-abstract parallel dataset. The cross-language document-abstract parallel dataset includes multiple source language document-target language reference abstract parallels. The source language document-target language reference abstract parallel includes a source language document and a target language reference abstract corresponding to the source language document;
[0106] The translation module 1002 processes each source language document-target language reference summary parallel data: machine translates the target language reference summary into a source language pseudo reference summary, and then machine translates the source language pseudo reference summary into a target language pseudo reference summary;
[0107] A similarity judgment module 1003 is used to judge the similarity between the target language pseudo reference summary of each source language document-target language reference summary parallel data and the target language reference summary of the source language document-target language reference summary parallel data, and based on the similarity, retain or discard the source language pseudo reference summary of the source language document-target language reference summary parallel data, and update the cross-language document-summary parallel data set; and
[0108] The encoder & decoder framework module 1004 inputs at least all the [source language document-target language reference summary-source language pseudo reference summary] triples in the updated cross-language document-summary parallel data set into the encoder & decoder framework module 1004 for training. After the training converges, a cross-language automatic summary generation model is obtained.
[0109] Figure 5 An exemplary diagram of an apparatus using a hardware implementation of a processing system is shown.
[0110] The cross-language automatic summary generation model building device 1000 may include corresponding modules for executing each or several steps in the above flowchart, such as a pre-processing module 1001 , a translation module 1002 , a similarity judgment module 1003 , and an encoder & decoder framework module 1004 .
[0111] Therefore, each step or several steps in the above flowchart can be performed by a corresponding module, and the device may include one or more modules in these modules. The module can be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.
[0112] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.
[0113] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.
[0114] Any process or method description in the flowchart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the present disclosure. The processor performs the various methods and processes described above. For example, the method implementation in the present disclosure can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).
[0115] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus, or device), or in combination with such instruction execution systems, apparatuses, or devices.
[0116] For the purposes of this specification, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples of readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a memory.
[0117] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0118] A person skilled in the art may understand that all or part of the steps of implementing the above-mentioned implementation method may be completed by instructing related hardware through a program, and the program may be stored in a readable storage medium, which, when executed, includes one or a combination of the steps of the implementation method.
[0119] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.
[0120] The present disclosure also provides an electronic device, including: a memory, the memory storing execution instructions; and a processor or other hardware module, the processor or other hardware module executes the execution instructions stored in the memory, so that the processor or other hardware module executes the above method.
[0121] The present disclosure also provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the above method.
[0122] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.
[0123] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0124] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.
Claims
1. A method for establishing a cross-language automatic summary generation model, characterized in that: include: S1. Preprocessing document-summary data in a cross-language document-summary parallel dataset, wherein the cross-language document-summary parallel dataset includes a plurality of source language document-target language reference summary parallel data, wherein the source language document-target language reference summary parallel data includes a source language document and a target language reference summary corresponding to the source language document; S2, processing each of the source language document-target language reference summary parallel data: machine translating the target language reference summary into a source language pseudo reference summary, and then machine translating the source language pseudo reference summary into a target language pseudo reference summary; S3, determining the similarity between the target language pseudo reference summary of each source language document-target language reference summary parallel data and the target language reference summary of the source language document-target language reference summary parallel data, and based on the similarity, retaining or discarding the source language pseudo reference summary of the source language document-target language reference summary parallel data, and updating the cross-language document-summary parallel data set; and S4, inputting at least all the [source language document-target language reference summary-source language pseudo reference summary] triplets in the updated cross-language document-summary parallel dataset into the encoder & decoder framework for training, and obtaining the cross-language automatic summary generation model after the training converges; The encoder & decoder framework includes a source language document encoder, a source language summary decoder, and a target language summary decoder, and different languages use independent vocabularies. In step S4, at least all the [source language document-target language reference summary-source language pseudo reference summary] triplets in the updated cross-language document-summary parallel dataset are input into the encoder & decoder framework for training, including: For each source language document in the [source language document-target language reference summary-source language pseudo-reference summary] triple, the source language-side summary decoder and the target language-side summary decoder synchronously perform summary decoding, and in the decoding process, the target language-side summary decoder fuses the source language document, the target language summary vocabulary generated by the target language-side summary decoder, and the source language summary vocabulary generated by the source language-side summary decoder at the decoding moment to predict the target language summary vocabulary at the decoding moment, and the source language-side summary decoder fuses the source language document, the target language summary vocabulary generated by the target language-side summary decoder, and the source language summary vocabulary generated by the source language-side summary decoder at the decoding moment to predict the source language summary vocabulary at the decoding moment.
2. The method for establishing a cross-language automatic summary generation model according to claim 1, characterized in that: Based on the similarity, retaining or discarding the source language pseudo reference summary of the source language document-target language reference summary parallel data includes: When the similarity is lower than a preset threshold, the source language pseudo reference summary of the source language document-target language reference summary parallel data is discarded; when the similarity is greater than or equal to the preset threshold, the source language pseudo reference summary of the source language document-target language reference summary parallel data is retained.
3. The method for establishing a cross-language automatic summary generation model according to claim 1, characterized in that: Based on the similarity, retaining or discarding the source language pseudo reference summary of the source language document-target language reference summary parallel data includes: When the similarity is lower than or equal to a preset threshold, the source language pseudo reference summary of the source language document-target language reference summary parallel data is discarded; when the similarity is greater than the preset threshold, the source language pseudo reference summary of the source language document-target language reference summary parallel data is retained.
4. The method for establishing a cross-language automatic summary generation model according to any one of claims 1 to 3, characterized in that: The similarity is the ROUGE indicator.
5. The method for establishing a cross-language automatic summary generation model according to any one of claims 1 to 3, characterized in that: The encoder & decoder framework adopts the Transformer architecture.
6. A cross-language automatic summary generation method, characterized in that: The cross-language automatic summary generation model established by the cross-language automatic summary generation model establishment method described in any one of claims 1 to 5 is used to synchronously decode the input source language document to generate a source language summary and a target language summary.
7. A device for establishing a cross-language automatic summary generation model, characterized in that: include: A preprocessing module, wherein the preprocessing module preprocesses document-summary data in a cross-language document-summary parallel dataset, wherein the cross-language document-summary parallel dataset includes a plurality of source language document-target language reference summary parallel data, wherein the source language document-target language reference summary parallel data includes a source language document and a target language reference summary corresponding to the source language document; A translation module, wherein the translation module processes each source language document-target language reference summary parallel data: machine translates the target language reference summary into a source language pseudo reference summary, and then machine translates the source language pseudo reference summary into a target language pseudo reference summary; a similarity judgment module, wherein the similarity judgment module judges the similarity between the target language pseudo reference summary of each source language document-target language reference summary parallel data and the target language reference summary of the source language document-target language reference summary parallel data, and based on the similarity, retains or discards the source language pseudo reference summary of the source language document-target language reference summary parallel data, and updates the cross-language document-summary parallel data set; and The encoder & decoder framework module inputs at least all the [source language document-target language reference summary-source language pseudo reference summary] triples in the updated cross-language document-summary parallel data set into the encoder & decoder framework module for training, and obtains the cross-language automatic summary generation model after the training converges; The encoder & decoder framework includes a source language document encoder, a source language summary decoder, and a target language summary decoder, and different languages use independent vocabularies. The step of inputting at least all the [source language document-target language reference summary-source language pseudo reference summary] triples in the updated cross-language document-summary parallel dataset into the encoder & decoder framework module for training comprises: For each source language document in the [source language document-target language reference summary-source language pseudo-reference summary] triple, the source language-side summary decoder and the target language-side summary decoder synchronously perform summary decoding, and in the decoding process, the target language-side summary decoder fuses the source language document, the target language summary vocabulary generated by the target language-side summary decoder, and the source language summary vocabulary generated by the source language-side summary decoder at the decoding moment to predict the target language summary vocabulary at the decoding moment, and the source language-side summary decoder fuses the source language document, the target language summary vocabulary generated by the target language-side summary decoder, and the source language summary vocabulary generated by the source language-side summary decoder at the decoding moment to predict the source language summary vocabulary at the decoding moment.
8. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor executes the method according to any one of claims 1 to 6.
9. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.