A method, device, and computer-readable storage medium for obtaining an abstract

Abstract

An embodiment of the present invention provides a method, device, and computer-readable storage medium for obtaining an abstract. The method includes: using a text encoder to encode a multimedia text to obtain text encoding features; using a title encoder to encode a multimedia title to obtain title encoding features; using a decoder to perform decoding at the current decoding time on the text encoding features and the title encoding features to obtain current decoding features, where the current decoding time is any time during the decoding process; using a word processing model to combine the text encoding features, the title encoding features, and the current decoding features to obtain a multi-source copy vocabulary distribution and a generation vocabulary distribution; and combining the multi-source copy vocabulary distribution and the generation vocabulary distribution, and determining a current keyword according to the combination result, so that when the decoding process ends, an objective abstract composed of multiple current keywords is obtained. Through the embodiments of the present invention, the accuracy and semantic richness of the obtained abstract can be improved.

Description

Technical Field

[0001] The present invention relates to the abstract acquisition technology in the field of artificial intelligence, and particularly to an abstract acquisition method, device and computer-readable storage medium. Background Art

[0002] An abstract, also known as a summary or synopsis, is the key points excerpted from a text. Generally, it refers to a short passage that concisely and accurately describes the important content of the text for the purpose of providing a general idea of the text content, without comments and supplementary explanations. Through the abstract, users can quickly obtain the main content of the text, and then be able to determine the target text they are interested in from a large amount of texts; therefore, it is crucial to determine the abstract of the text.

[0003] Generally speaking, when obtaining the abstract of a text, it is usually achieved by the abstract generation method or the abstract extraction method. However, for the abstract of the text obtained by the abstract generation method, there are problems such as unsmooth sentences or sensitive words and other insecure generation problems; while for the abstract of the text obtained by the abstract extraction method, due to the source of information being only limited to the text itself, there is a problem of single information; therefore, the accuracy of the obtained abstract is poor or the semantic richness is low. Summary of the Invention

[0004] Embodiments of the present invention provide an abstract acquisition method, device and computer-readable storage medium, which can improve the accuracy and semantic richness of the obtained abstract.

[0005] The technical solution of the embodiments of the present invention is implemented as follows:

[0006] Embodiments of the present invention provide an abstract acquisition method, including:

[0007] Obtain a multimedia text, and obtain a preset abstract acquisition model; the multimedia text includes a multimedia body and a multimedia title, and the preset abstract acquisition model includes a body encoder, a title encoder, a decoder and a word processing model;

[0008] Use the body encoder to encode the multimedia body to obtain a body encoding feature; and use the title encoder to encode the multimedia title to obtain a title encoding feature;

[0009] Use the decoder to perform decoding at the current decoding moment on the body encoding feature and the title encoding feature to obtain a current decoding feature; the current decoding moment is any moment of the decoding moment;

[0010] Using the word processing model, combine the body text encoding feature, the title encoding feature, and the current decoding feature to obtain a multi-source copy vocabulary distribution and a generated vocabulary distribution; and combine the multi-source copy vocabulary distribution and the generated vocabulary distribution, and determine the current keyword according to the combination result, so that when the decoding moment ends, obtain a target summary composed of multiple current keywords;

[0011] Among them, the multi-source copy vocabulary distribution is the distribution information of keywords in the multimedia text, and the generated vocabulary distribution is the distribution information of keywords generated based on the multimedia text.

[0012] An embodiment of the present invention provides a summary acquisition device, including:

[0013] An acquisition module, configured to acquire a multimedia text and acquire a preset summary acquisition model; the multimedia text includes a multimedia body text and a multimedia title, and the preset summary acquisition model includes a body text encoder, a title encoder, a decoder, and a word processing model;

[0014] An encoding module, configured to use the body text encoder to encode the multimedia body text to obtain a body text encoding feature; and use the title encoder to encode the multimedia title to obtain a title encoding feature;

[0015] A decoding module, configured to use the decoder to perform decoding at the current decoding moment on the body text encoding feature and the title encoding feature to obtain a current decoding feature; the current decoding moment is any moment of the decoding moment;

[0016] A summary acquisition module, configured to use the word processing model to combine the body text encoding feature, the title encoding feature, and the current decoding feature to obtain a multi-source copy vocabulary distribution and a generated vocabulary distribution; and combine the multi-source copy vocabulary distribution and the generated vocabulary distribution, and determine the current keyword according to the combination result, so that when the decoding moment ends, obtain a target summary composed of multiple current keywords; among them, the multi-source copy vocabulary distribution is the distribution information of keywords in the multimedia text, and the generated vocabulary distribution is the distribution information of keywords generated based on the multimedia text.

[0017] An embodiment of the present invention provides a summary acquisition device, including:

[0018] A memory, configured to store executable instructions;

[0019] A processor, configured to implement the summary acquisition method provided by the embodiment of the present invention when executing the executable instructions stored in the memory.

[0020] An embodiment of the present invention provides a computer-readable storage medium storing executable instructions, which are used to cause a processor to execute and implement the abstract acquisition method provided by the embodiment of the present invention.

[0021] The embodiment of the present invention has the following beneficial effects: When obtaining the target abstract corresponding to the multimedia text, the model adopted determines the probability value of the decoded keyword from two aspects: the keyword in the multimedia text and the keyword generated based on the multimedia text; as a result, the obtained target abstract not only has the accuracy of the abstract obtained by the abstract extraction method but also has the rich semantics of the abstract obtained by the abstract generation method; in addition, both the text body and the title of the multimedia text are used as the copy sources of the decoded keywords, implementing the technical solution of using the title of the multimedia text as guiding information to extract keywords from the multimedia text as part of the information of the target abstract, and the title of the multimedia text can accurately express the important meaning of the multimedia text. Therefore, the accuracy of the obtained target abstract is further improved; thus, the accuracy and semantic richness of the obtained abstract are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is an exemplary application schematic diagram of abstract acquisition;

[0023] Figure 2 is an optional architecture schematic diagram of the abstract acquisition system 100 provided by the embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of the structure of the server 200 provided by the embodiment of the present invention;

[0025] Figure 4 is an optional flowchart of the abstract acquisition method provided by the embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of the structure of the preset abstract acquisition model provided by the embodiment of the present invention;

[0027] Figure 6 is another optional flowchart of the abstract acquisition method provided by the embodiment of the present invention;

[0028] Figure 7 is an abstract acquisition architecture schematic diagram provided by the embodiment of the present invention;

[0029] Figure 8 is another optional flowchart of the abstract acquisition method provided by the embodiment of the present invention;

[0030] Figure 9 is another optional architecture schematic diagram of the abstract acquisition system 100 provided by the embodiment of the present invention;

[0031] Figure 10 It is a schematic structural diagram of a blockchain in the blockchain network 600 provided by an embodiment of the present invention;

[0032] Figure 11 It is a schematic functional architecture diagram of the blockchain network 600 provided by an embodiment of the present invention;

[0033] Figure 12 It is a schematic diagram of the application of an exemplary abstract acquisition method provided by an embodiment of the present invention;

[0034] Figure 13 It is a schematic diagram of displaying a target abstract provided by an embodiment of the present invention. Detailed implementation manners

[0035] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be construed as limitations on the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0036] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0037] In the following description, the terms "first / second" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second" can be interchanged with a specific order or sequence when allowed, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0038] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present invention have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the embodiments of the present invention are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0039] Before further elaborating on the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following explanations.

[0040] 1) Artificial Intelligence (AI) is a theory, method, technology and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0041] 2) Machine Learning (ML) is an interdisciplinary subject that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills, and reorganize the existing knowledge structure to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications cover all fields of artificial intelligence. Machine learning usually includes technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0042] 3) An artificial neural network is a mathematical model that mimics the structure and function of a biological neural network. The exemplary structures of artificial neural networks in this article include Deep Neural Networks (DNN), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN). Among them, the neural network layers inside DNN can be divided into three categories: the input layer, the hidden layer, and the output layer, and they are fully connected between layers. That is to say, any neuron in the k-th layer must be connected to any neuron in the k+1-th layer; in the field of abstract acquisition, it can reasonably infer and derive the interests of users and improve content diversity.

[0043] 4) A loss function, also known as a cost function, is a function that maps the values of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of the random event.

[0044] 5) Sequence to Sequence (seq2seq). In the sequence-to-sequence model, through two neural networks, one neural network (Encoder) is responsible for receiving multimedia text, and the other neural network (Decoder) is responsible for outputting the multimedia text into a target text of another sequence (for example, the abstract of multimedia text). These two processes are called the encoding process and the decoding process respectively. Among them, in the encoding process, Encoder encodes the signal sequence of variable-length multimedia text into a fixed-length vector representation for easy mathematical processing; in the decoding process, Decoder decodes the fixed-length vector into a variable-length target signal sequence.

[0045] 6) The Attention mechanism refers to focusing attention on important points while ignoring other unimportant factors. In the seq2seq model, during the Encoder process of the original encoding and decoding model, an intermediate vector H is generated to preserve the semantic information of the multimedia text. However, the length of this intermediate vector H is fixed. When the length of the multimedia text sequence is relatively long, the intermediate vector H cannot preserve all the semantic information of the multimedia text, and the context semantic information of the multimedia text is restricted, thus limiting the processing ability of the model. Therefore, the Attention mechanism is used to solve the limitation of the encoding and decoding model on the fixed length.

[0046] 7) Blockchain is an encrypted, chain-like storage structure of transactions formed by blocks.

[0047] 8) A blockchain network is a collection of a series of nodes that incorporate new blocks into the blockchain through a consensus method.

[0048] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in multiple fields; for example, common ones such as smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless, autonomous driving, drones, robots, smart healthcare, and smart customer service, etc. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role; for instance, it can also be applied in the field of abstract acquisition.

[0049] Here, it should be noted that artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence is also about researching the design principles and implementation methods of various intelligent machines to enable the machines to have the functions of perception, reasoning, and decision-making.

[0050] In addition, artificial intelligence technology is an interdisciplinary subject involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics, etc. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0051] It should be noted that the abstract of text is a basic technology in the field of natural language processing and is usually applied in application scenarios such as recommendation. In addition, when obtaining the abstract of text, it is divided into multi-document abstract acquisition and single-document abstract acquisition according to the number of documents in the text. The abstract acquisition method in the embodiments of the present invention is a single-document abstract acquisition method.

[0052] Generally speaking, when obtaining the abstract of multimedia text, it is usually implemented by the abstract generation method (Abstrac tive) or the abstract extraction method (Extractive). Among them, the abstract generation method refers to the method of generating a new word or phrase according to the multimedia text to form an abstract. The abstract obtained by this method is called a generative abstract; for example, ConvS2S (Convolutional Sequence to Sequence) uses a convolutional neural network to extract the N-Gram features of sentences, introduces an Att ention mechanism between the Encoder and the Decoder, and uses beam search to decode and generate words in the vocabulary, and finally forms the abstract of the complete text. When using the abstract generation method to obtain the abstract of multimedia text, although the information of the obtained abstract is diverse, the content is rich, and the creativity is good, it is necessary to solve the problems of the representation and application of OOV (Out of Word, out-of-vocabulary words). In addition, in the abstract of the generated text, there are unsafe generation problems such as unsmooth sentences or sensitive words.

[0053] The abstract extraction method refers to the process of extracting information from multimedia text to form an abstract. The abstract obtained by this method is called an extractive abstract; moreover, corresponding to the abstract extraction method, it is divided into a word extraction method and a sentence extraction method based on different extraction units. For the word extraction method, for example, CopyNet first calculates the probability value of words in the multimedia text, and then copies words from the multimedia text according to the probability value to form an abstract. For the sentence extraction method, for example, Summarunner represents the sentences in the multimedia text as feature vectors, calculates the score value of each sentence according to the feature vector of each sentence, and then selects whether to extract the sentence according to the score value, and finally combines the extracted sentences to form an abstract. When using the abstract extraction method to obtain the abstract of multimedia text, although the accuracy of the obtained abstract is high and there is no unsafe generation problem, since the abstract is the information extracted from the multimedia text, the information corresponding to the abstract is single, the content is simple, the creativity is low, and there is no feature.

[0054] In summary, the semantic expression ability of the generative summary is strong, but there is a problem of insecure generation; while the extractive summary has high accuracy, but there is a problem of low semantic expression ability; when obtaining the summary of multimedia text, combine the summary generation method and the summary extraction method to obtain the summary; for example, the Pointer Generator, see Figure 1 , after the multimedia text 1-1 is encoded by the input encoder Encoder1-2, the encoded result is input into the decoder 1-3; taking the second decoding moment of the decoder 1-3 as an example for illustration, at this time, the decoding output state 1-31 is matched with the output of the encoding to obtain the attention weight 1-4 of the multimedia text 1-1, that is, the weight of the Decoder reading information from the Encoder calculated by using the Pointer Networks; based on the output of the encoding and the attention weight 1-4, the context vector 1-5 of the multimedia text 1-1 is obtained; based on the context vector 1-5 and the decoding output state 1-31, the probability distribution 1-6 of the generated vocabulary is obtained; in addition, according to the context vector 1-5 of the multimedia text and the decoding output state 1-31, the distribution weight p gen (belonging to 0 to 1) is determined, and based on p gen Combining the attention weight 1-4 of the multimedia text and the probability distribution 1-6 of the generated vocabulary, an extended vocabulary probability distribution is obtained, and then the current predicted word 1-7 is determined according to the extended vocabulary probability distribution, and finally the summary 1-8 of the multimedia text is obtained. In this way, the method of combining the summary generation method and the summary extraction method to obtain the summary is completed, and thus, the obtained summary has accuracy and strong expression ability. However, in the process of obtaining the summary by combining the summary generation method and the summary extraction method as described above, there is a problem of inaccurate focus on the multimedia text. For example, some important information is not focused on, and the copied words and information are unimportant, resulting in a low accuracy rate of the obtained summary; in addition, there is also a situation where important words (such as personal names, place names, and other important descriptive words) are missing in both the text and the generated vocabulary of the multimedia text, which also leads to the problem of low accuracy rate of the obtained summary. In summary, the accuracy of the obtained summary is low, and thus the acquisition effect of the summary is poor.

[0055] Based on this, the embodiments of the present invention provide a summary acquisition method, device and computer-readable storage medium, which can improve the acquisition effect of the summary and improve the accuracy of the obtained summary. The following describes an exemplary application of the summary acquisition device provided by the embodiments of the present invention. The summary acquisition device provided by the embodiments of the present invention can be implemented as various types of user terminals such as smart phones, tablet computers, and laptop computers, or can also be implemented as a server. Next, an exemplary application when the summary acquisition device is implemented as a server will be described.

[0056] SeeFigure 2 , Figure 2 is an optional schematic architecture diagram of the abstract acquisition system 100 provided by an embodiment of the present invention. To support an abstract acquisition application, a server 400 (multimedia server) is connected to a database 500 (a general term for a preset task queue and a preset abstract cache) through a network 300. The database 500 is the database corresponding to a server 200 (abstract acquisition device). The network 300 can be a wide area network, a local area network, or a combination of both. In addition, the above-mentioned abstract acquisition system 100 further includes a terminal 410.

[0057] The server 200 is configured to obtain multimedia text from the database 500 and obtain a preset abstract acquisition model. The multimedia text includes a multimedia body and a multimedia title. The preset abstract acquisition model includes a body encoder, a title encoder, a decoder, and a word processing model. The body encoder is used to encode the multimedia body to obtain a body encoding feature. The title encoder is used to encode the multimedia title to obtain a title encoding feature. The decoder is used to decode the body encoding feature and the title encoding feature at the current decoding moment to obtain a current decoding feature. The current decoding moment is any moment of the decoding moment. The word processing model is used to combine the body encoding feature, the title encoding feature, and the current decoding feature to obtain a multi-source copy vocabulary distribution and a generated vocabulary distribution. The multi-source copy vocabulary distribution and the generated vocabulary distribution are combined, and the current keyword is determined according to the combination result. Thus, when the decoding moment ends, a target abstract composed of multiple current keywords is obtained. Among them, the multi-source copy vocabulary distribution is the distribution information of keywords in the multimedia text, and the generated vocabulary distribution is the distribution information of keywords generated based on the multimedia text.

[0058] The terminal 410 is configured to obtain the multimedia text and the target abstract from the database 500 through the network 300 and the server 400, and display the multimedia text and the target abstract in a graphical interface 411.

[0059] It should be noted that the embodiments of the present invention can also be implemented in combination with blockchain technology. Blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, a string of data blocks generated by using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of the information (anti-counterfeiting) and generate the next block. Blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer. For the abstract acquisition method provided by the embodiments of the present invention in combination with blockchain technology, please refer to the subsequent description.

[0060] See Figure 3 , Figure 3It is a schematic structural diagram of the server 200 provided by an embodiment of the present invention. Figure 3 The shown server 200 includes: at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. Each component in the server 200 is coupled together through a bus system 240. It can be understood that the bus system 240 is used to implement the connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 3 all kinds of buses are labeled as the bus system 240.

[0061] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] The user interface 230 includes one or more output devices 231 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that assist the user in input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons, and controls.

[0063] The memory 250 includes volatile memory or non-volatile memory, and may also include both volatile and non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM, Read Only Memory), and the volatile memory can be a random access memory (RAM, Random Access Memory). The memory 250 described in the embodiments of the present invention is intended to include any suitable type of memory. Optionally, the memory 250 includes one or more storage devices that are physically located far from the processor 210.

[0064] In some embodiments, the memory 250 is capable of storing data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof, which are exemplarily described below.

[0065] An operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0066] A network communication module 252 for reaching other computing devices via one or more (wired or wireless) network interfaces 220. Exemplary network interfaces 220 include: Bluetooth, Wi-Fi (Wireless Fidelity), and USB (Universal Serial Bus), etc.;

[0067] A display module 253 for enabling the presentation of information (such as a user interface for operating peripheral devices and displaying content and information) via one or more output devices 231 associated with the user interface 230 (such as a display screen, a speaker, etc.);

[0068] An input processing module 254 for detecting and translating one or more user inputs or interactions from one of one or more input devices 232.

[0069] In some embodiments, the abstract acquisition device provided by the embodiments of the present invention may be implemented in a software manner. Figure 3 It shows an abstract acquisition device 255 stored in the memory 250, which may be software in the form of a program and a plug-in, etc., including the following software modules: an acquisition module 2551, an encoding module 2552, a decoding module 2553, an abstract acquisition module 2554, a model training module 2555, a storage module 2556, and a blockchain module 2557. The functions of each module will be described below.

[0070] In other embodiments, the abstract acquisition device provided by the embodiments of the present invention may be implemented in a hardware manner. As an example, the abstract acquisition device provided by the embodiments of the present invention may be a processor in the form of a hardware decoding processor, which is programmed to execute the abstract acquisition method provided by the embodiments of the present invention. For example, a processor in the form of a hardware decoding processor may employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0071] Next, the abstract acquisition method provided by the embodiments of the present invention will be described in conjunction with the exemplary applications and implementations of the server provided by the embodiments of the present invention.

[0072] See Figure 4 , Figure 4FIG. 0 is an optional flowchart of the abstract acquisition method provided by an embodiment of the present invention, which will be described in conjunction with Figure 4 the steps shown.

[0073] S101. Obtain a multimedia text and obtain a preset abstract acquisition model; the multimedia text includes a multimedia body and a multimedia title, and the preset abstract acquisition model includes a body encoder, a title encoder, a decoder, and a word processing model.

[0074] In an embodiment of the present invention, when the abstract acquisition device performs abstract acquisition, the object it targets is the multimedia text, for example, news information; and when the abstract acquisition device receives the multimedia text or reads the multimedia text from a preset storage device, it obtains the multimedia text.

[0075] It should be noted that the multimedia text includes a multimedia body and a multimedia title; among them, the multimedia body refers to the body of the multimedia text, and the multimedia title refers to the title of the multimedia text.

[0076] In addition, a trained model for obtaining the abstract of the multimedia text, that is, a preset abstract acquisition model, is pre-stored in the abstract acquisition device, for example, the MSPG model, or the MSPG model introduced with the coverage mechanism; therefore, the abstract acquisition device can obtain the preset abstract acquisition model. And the preset abstract acquisition model is composed of multiple sub-models. Here, the preset abstract acquisition model includes a body encoder, a title encoder, a decoder, and a word processing model.

[0077] It should be noted that the body encoder is used to encode the multimedia body to obtain a semantic vector representation of the multimedia body, for example, a unidirectional LSTM (Long Short-Term Memory), a bidirectional LSTM, a feature extractor Transformer; the title encoder is used to encode the multimedia title to obtain a semantic feature representation of the multimedia title, for example, a unidirectional LSTM, a bidirectional LSTM, a feature extractor Transformer; the decoder is used to decode the encoding results of the body encoder and the title encoder to obtain another information representation of the multimedia text, for example, a unidirectional LSTM, a bidirectional LSTM; the word processing model is used to process the semantic feature representation of the multimedia body, the semantic feature representation of the multimedia title, and the another information representation of the multimedia text obtained by decoding to determine the abstract of the multimedia text.

[0078] S102. Use the body encoder to encode the multimedia body to obtain body encoding features; and use the title encoder to encode the multimedia title to obtain title encoding features.

[0079] In an embodiment of the present invention, after the abstract acquisition device obtains a preset abstract acquisition model and a multimedia text, it can use the preset abstract acquisition model to obtain the abstract of the multimedia text. Here, the abstract acquisition device first extracts the semantic features of the multimedia text using a text encoder, that is, encodes the multimedia text to obtain a semantic feature representation of the multimedia text, and thus obtains a text encoding feature; and extracts the semantic features of the media title using a title encoder, that is, encodes the multimedia title to obtain a semantic feature representation of the multimedia title, and thus obtains a title encoding feature.

[0080] It should be noted that when the abstract acquisition device encodes the multimedia text using the text encoder, it first performs a vector representation on the multimedia text, and then uses the text encoder to extract the semantic features of the vector-represented multimedia text; similarly, when the abstract acquisition device encodes the media title using the title encoder, it also first performs a vector representation on the multimedia title, and then uses the title encoder to extract the semantic features of the vector-represented multimedia title. In addition, when the abstract acquisition device performs a vector representation on the multimedia text and the multimedia title, it can also be implemented based on enhanced semantic background knowledge and / or an optimized initial word vector method.

[0081] S103. Use a decoder to decode the text encoding feature and the title encoding feature at the current decoding time to obtain a current decoding feature; the current decoding time is any time of the decoding time.

[0082] In an embodiment of the present invention, after the abstract acquisition device obtains the text encoding feature and the title encoding feature, since the text encoding feature characterizes the semantic features of the multimedia text and the title encoding feature characterizes the semantic features of the multimedia title; therefore, the abstract acquisition device can use the decoder to represent the multimedia text as another information representation for the text encoding feature and the title encoding feature at each decoding time; here, for any time of the decoding time, the other information representation of the decoded multimedia text is the current decoding feature.

[0083] It should be noted that when the decoder decodes, it can be based on a preset word list or a dynamic word list, and the embodiments of the present invention do not make specific limitations on this.

[0084] S104. Use a word processing model to combine the text encoding feature, the title encoding feature, and the current decoding feature to obtain a multi-source copy word list distribution and a generated word list distribution; and combine the multi-source copy word list distribution and the generated word list distribution, and determine the current keyword according to the combination result, so that when the decoding time ends, a target abstract composed of multiple current keywords is obtained; where the multi-source copy word list distribution is the distribution information of the keywords in the multimedia text, and the generated word list distribution is the distribution information of the keywords generated based on the multimedia text.

[0085] In an embodiment of the present invention, the body encoding feature, the title encoding feature, and the current decoding feature are used to determine the keyword corresponding to the current decoding moment; therefore, after the abstract acquisition device obtains the body encoding feature, the title encoding feature, and the current decoding feature, it can use the word processing model to process the body encoding feature, the title encoding feature, and the current decoding feature to determine the keyword corresponding to the current decoding moment.

[0086] Here, the abstract acquisition device combines the body encoding feature, the title encoding feature, and the current decoding feature by using the word processing model, and can obtain the distribution information of the keywords in the multimedia text, that is, the multi-source copy word list distribution; and can also obtain the distribution information of the keywords generated based on the multimedia text, that is, the generated word list distribution. Here, the generated word list distribution refers to the probability value of each keyword in the preset word list or the dynamic word list corresponding to the output at the current decoding moment; and the preset word list is composed of the words obtained by segmenting the preset corpus, sorting the segmentation results in reverse order based on the word frequency, and then selecting the first preset number of words from the sorted segmentation results.

[0087] Since the multi-source copy word list distribution represents the probability distribution that the keyword output at the current decoding moment comes from the multimedia text, and the generated word list distribution represents the probability distribution that the keyword output at the current decoding moment is a generated word; therefore, the abstract acquisition device combines the multi-source copy word list distribution and the generated word list distribution by using the word processing model, and selects the preset number of keywords with the largest probabilities from the combined result as the keywords output at the current decoding moment, and thus obtains the current keyword; at this time, the decoding of the current decoding moment is completed.

[0088] In an embodiment of the present invention, the abstract acquisition device performs the operations corresponding to the current decoding moment for any moment in the decoding moments, and thus obtains multiple current keywords, and further can obtain the abstract of the multimedia text, that is, the target abstract, according to the multiple current keywords.

[0089] It should be noted that when the abstract acquisition device obtains the target abstract according to the multiple current keywords, it comprehensively considers the context relationship between the multiple sub-current keywords in each current keyword and the multiple sub-current keywords in other current keywords among the multiple current keywords, and selects one sub-current keyword from each current keyword to form the target abstract to determine that the obtained target abstract is the optimal solution. For example, the abstract acquisition device uses the "Beam Search" algorithm to determine the target abstract according to the multiple current keywords.

[0090] It can be understood that the abstract acquisition method provided by the embodiments of the present invention combines the respective advantages of the abstract generation method and the abstract extraction method, making the target abstract of the obtained multimedia text accurate and semantically rich. At the same time, since the preset abstract acquisition model adopted copies keywords from the text and title of the multimedia text as candidate words for the target abstract, and uses the encoding features of the text and title of the multimedia text as basic information for generating keywords; that is, the title of the multimedia text is used as guiding information for separate encoding and copying words, which enables accurate focus on the important information of the multimedia text, avoids the problem of cumulative errors, and improves the accuracy and semantic richness of the target abstract. In addition, the encoder in the preset abstract acquisition model can be two or more, and the model structure has scalability.

[0091] Further, in the embodiments of the present invention, the word processing model includes a contribution model, a context vector model, a word copy model, and a word generation model; wherein, the contribution model is used to determine the information contribution degree of the encoding result at each moment in the preset abstract acquisition model at the current decoding moment; the context vector model is used to determine the information of the multimedia text on which the decoder depends when decoding at the current decoding moment; the word copy model is used to determine the probability distribution of the keywords copied from the multimedia text corresponding to the current decoding moment; the word generation model is used to determine the probability distribution of the generated keywords corresponding to the current decoding moment. Thus, the abstract acquisition device in step S104 of the embodiments of the present invention uses the word processing model to match and combine the text encoding feature, the title encoding feature, and the current decoding feature, and obtains the multi-source copy word list distribution and the generated word list distribution, including S1041 - S1044, which will be described below in combination with each step.

[0092] S1041. Use the contribution model to perform first normalization processing on the combination of the text encoding feature and the current decoding feature, and the combination of the title encoding feature and the current decoding feature respectively, to obtain the text information contribution degree corresponding to the text encoding feature, and the title information contribution degree corresponding to the title encoding feature.

[0093] In the embodiments of the present invention, since the current decoding feature is obtained based on the text encoding feature and the title encoding feature; therefore, in order for the abstract acquisition device to determine the information contribution degree of the text encoding feature to the current decoding feature, it calculates the matching degree between the text encoding feature and the current decoding feature, and thus obtains the information contribution degree of the text encoding feature to the current decoding feature, and also obtains the text information contribution degree corresponding to the text encoding feature; similarly, in order for the abstract acquisition device to determine the information contribution degree of the title encoding feature to the current decoding feature, it calculates the matching degree between the title encoding feature and the current decoding feature, and thus obtains the information contribution degree of the title encoding feature to the current decoding feature, and also obtains the title information contribution degree corresponding to the title encoding feature.

[0094] Here, the abstract acquisition device uses the contribution degree model to perform the first normalization process on the combination of the text encoding feature and the current decoding feature, and thus obtains the text information contribution degree; similarly, the abstract acquisition device uses the contribution degree model to perform the first normalization process on the combination of the title encoding feature and the current decoding feature, and thus obtains the title information contribution degree. Here, the first normalization process is used to determine the matching degree.

[0095] Exemplarily, the contribution degree model is shown in equations (1) and (2):

[0096]

[0097] a t = softmax(e t ) (2)

[0098] where, v T , W h , W s and b attn are the training parameters in the contribution degree model, tanh() and softmax() are both existing function operations, h i is the encoding feature at the i-th encoding moment in the encoding feature (text encoding feature or title encoding feature), s t is the decoding feature (current decoding feature) at the t-th moment (current decoding moment); is the contribution degree information of the encoding feature at the i-th encoding moment to the decoding feature at the t-th moment; e t is the set composed of the contribution degree information of all encoding moments; a t is the set composed of the contribution degrees corresponding to the encoding features of all encoding moments respectively, that is, the information contribution degree (text information contribution degree or title information contribution degree).

[0099] S1042. Use the context vector model to combine the text encoding feature with the text information contribution degree to obtain the text context vector; and combine the title encoding feature with the title information contribution degree to obtain the title context vector.

[0100] In an embodiment of the present invention, after the summary acquisition device obtains the contribution of the main text information and the contribution of the title information, it uses the context vector model to fuse the sub-contributions in the main text information contribution with the coding features of each coding moment in the main text coding features in a one-to-one correspondence (weighted summation after corresponding combination, or average after corresponding combination, etc.), and the obtained fusion result is the information of the multimedia main text based on which the decoding is performed at the current decoding moment, and the main text context vector is obtained; similarly, the summary acquisition device uses the context vector model to fuse the sub-contributions in the title information contribution with the coding features of each coding moment in the title coding features in a one-to-one correspondence, and the obtained fusion result is the information of the multimedia title based on which the decoding is performed at the current decoding moment, and the title context vector is obtained.

[0101] It should be noted that the text context vector is a fixed-length representation of the information read by the decoder from the semantic features of the multimedia text at the current decoding moment; the title context vector is a fixed-length representation of the information read by the decoder from the semantic features of the multimedia title at the current decoding moment.

[0102] For example, the contribution model is shown in formula (3):

[0103]

[0104] in, is the context vector (text context vector or title context vector), Refers to a in formula (2) t The information contribution of the i-th encoding moment, h i is the coding feature of the i-th coding moment in the coding feature (text coding feature or title coding feature), n is the number of coding moments (the number of coding moments of the multimedia text or the number of coding moments of the multimedia title, 1 to n is the value range of i).

[0105] S1043. Using the word replication model, based on the current decoding features, the contribution of the text information and the contribution of the title information are integrated to obtain a multi-source replication word list distribution.

[0106] In an embodiment of the present invention, after the summary acquisition device obtains the current decoding features, the text information contribution and the title information contribution, it can use the word copying model to fuse the text information contribution and the title information contribution based on the current decoding features, and obtain the probability distribution of keywords copied from the multimedia text, thereby obtaining the multi-source copied word list distribution.

[0107] S1044: Using the word generation model, perform a second normalization process on the text context vector, the title context vector, and the current decoding feature to obtain a generated word list distribution.

[0108] In an embodiment of the present invention, after the summary acquisition device obtains the text context vector and the title context vector, it performs a second normalization process in combination with the current decoding features, thereby obtaining the probability distribution of each keyword in the preset vocabulary at the current decoding moment, and thus obtaining the generated vocabulary distribution.

[0109] For example, the word generation model is shown in formula (4):

[0110]

[0111] Among them, P vocab To generate the word list distribution, softmax() is an existing function operation, V', V', b and b' are the training parameters in the word generation model, s t is the decoding feature (current decoding feature) at time t (current decoding time), is the text context vector, is the title context vector, For s t 、 and Here, it is easy to know that and It is obtained according to formula (1)-(3).

[0112] It should be noted that there is no particular order in which S1043 and S1044 are executed.

[0113] Furthermore, in an embodiment of the present invention, the word copying model includes a word determination model, a text word distribution model, a title word distribution model and a copy distribution combination model; wherein the word determination model is used to determine the keywords to be copied from the multimedia text; the text word distribution model is used to determine the probability distribution of the keywords to be copied in the multimedia text based on the current decoding features and the text information contribution; the title word distribution model is used to determine the probability distribution of the keywords to be copied in the multimedia title based on the current decoding features and the title information contribution; the copy distribution combination model is used to combine the probability distribution of the keywords to be copied in the multimedia text and the probability distribution of the keywords to be copied in the multimedia title; thus, S1043 can be implemented through S10431-S10434; that is, the summary acquisition device uses the word copying model to fuse the text information contribution and the title information contribution based on the text context vector and the title context vector to obtain a multi-source copy word list distribution, including S10431-S10434, which will be explained in conjunction with each step below.

[0114] S10431. Use the word determination model to copy keywords from the multimedia text and the multimedia title respectively, to obtain a text copy word list corresponding to the multimedia text and a title copy word list corresponding to the multimedia title.

[0115] In an embodiment of the present invention, the abstract acquisition device uses the word determination model to determine the keywords to be copied from the multimedia text, and the determined keywords to be copied constitute the text copy word list; similarly, the abstract acquisition device uses the word determination model to determine the keywords to be copied from the multimedia title, and the determined keywords to be copied constitute the title copy word list; at this time, the process of copying keywords from the multimedia text and the multimedia title respectively, to obtain a text copy word list corresponding to the multimedia text and a title copy word list corresponding to the multimedia title is completed.

[0116] S10432. Use the text word distribution model to determine a target text word list that matches the current decoding feature from the text copy word list, and determine the target text information contribution degree corresponding to each keyword in the target text word list from the text information contribution degree, and perform a weighted sum of the target text information contribution degrees to obtain a text copy word list distribution corresponding to the text copy word list.

[0117] In an embodiment of the present invention, the abstract acquisition device uses the text word distribution model to form a target text word list with the keywords in the text copy word list that match the current decoding feature, and determines the target text information contribution degree corresponding to each keyword in the target text word list from the text information contribution degree, and performs a weighted sum of the target text information contribution degrees, thus obtaining the probability distribution of the keywords to be copied in the multimedia text, and obtaining a text copy word list distribution corresponding to the text copy word list.

[0118] S10433. Use the title word distribution model to determine a target title word list that matches the current decoding feature from the title copy word list, and determine the target title information contribution degree corresponding to each keyword in the target title word list from the title information contribution degree, and perform a weighted sum of the target title information contribution degrees to obtain a title copy word list distribution corresponding to the title copy word list.

[0119] In an embodiment of the present invention, the abstract acquisition device uses the title word distribution model to form a target title word list with the keywords in the title copy word list that match the current decoding feature, and determines the target title information contribution degree corresponding to each keyword in the target title word list from the title information contribution degree, and performs a weighted sum of the target title information contribution degrees, thus obtaining the probability distribution of the keywords to be copied in the multimedia title, and obtaining a title copy word list distribution corresponding to the title copy word list.

[0120] S10434. Combine the text copy vocabulary distribution and the title copy vocabulary distribution using a copy distribution combination model to obtain a multi-source copy vocabulary distribution.

[0121] In an embodiment of the present invention, after the abstract acquisition device obtains the text copy vocabulary distribution and the title copy vocabulary distribution, it combines the text copy vocabulary distribution and the title copy vocabulary distribution, thereby obtaining a multi-source copy vocabulary distribution.

[0122] Further, in an embodiment of the present invention, the copy distribution combination model includes a decoding input acquisition model, a copy weight determination model, and a multi-source copy model. The decoding input acquisition model is used to obtain the input features at the current decoding moment. The copy weight determination model is used to determine the weight of the text copy vocabulary distribution in the multi-source copy vocabulary distribution and the weight of the title copy vocabulary distribution in the multi-source copy vocabulary distribution. The multi-source copy model is used to determine the result after the fusion of the multiple copy vocabulary distributions (the text copy vocabulary distribution and the title copy vocabulary distribution) based on the multiple copy vocabulary distributions. Thus, S10434 can be implemented through S104341 - S104343. That is to say, the abstract acquisition device uses the copy distribution combination model to combine the text copy vocabulary distribution and the title copy vocabulary distribution to obtain a multi-source copy vocabulary distribution, including S104341 - S104343, which will be described in conjunction with each step below:

[0123] S104341. Use the decoding input acquisition model to obtain the current decoding input features at the current decoding moment.

[0124] In an embodiment of the present invention, when the decoder performs decoding at the current moment, input features are required. Therefore, the abstract acquisition device can use the input acquisition model to obtain the input features at the current decoding moment, thereby obtaining the current decoding input features.

[0125] It should be noted that the current decoding input features are the vector representation of the decoding result at the previous decoding moment. For example, the Embedding value of the decoding result at the previous decoding moment. When the current decoding moment is the first decoding moment, the current decoding input features are the encoding results of both the text encoder and the title encoder.

[0126] S104342. Use the copy weight determination model to perform a first non-linear combination on the text context vector, the title context vector, the current decoding state, and the current decoding input features to obtain the weight of the text copy vocabulary distribution, and obtain the opposite value of the weight of the text copy vocabulary distribution to obtain the weight of the title copy vocabulary distribution.

[0127] In an embodiment of the present invention, after the abstract acquisition device obtains the body context vector, the title context vector, the current decoding state, and the current decoding input feature, it can use the copy weight determination model to perform a first non-linear combination on the body context vector, the title context vector, the current decoding state, and the current decoding input feature. The obtained result is the weight of the body copy vocabulary distribution in the multi-source copy vocabulary distribution, that is, the weight of the body copy vocabulary distribution is obtained. At this time, by obtaining the opposite value of the weight of the body copy vocabulary distribution, the weight of the title copy vocabulary distribution in the multi-source copy vocabulary distribution is obtained, that is, the weight of the title copy vocabulary distribution is obtained.

[0128] It should be noted that the process of obtaining the opposite value is relative to a numerical benchmark. For example, when the numerical benchmark is 1 and the weight of the body copy vocabulary distribution is λ, the opposite value of the weight of the body copy vocabulary distribution is 1 - λ.

[0129] Exemplarily, the copy weight determination model is as shown in Equation (5):

[0130]

[0131] where λ is the weight of the body copy vocabulary distribution, σ() is an existing functional operation, and b1 are both training parameters in the copy weight determination model, is the body context vector, is the title context vector, s t is the decoding feature (current decoding feature) at time t (current decoding time), x t is the input feature (current decoding input feature) at time t (current decoding time). Here, it is easy to know that the weight of the title copy vocabulary distribution is 1 - λ.

[0132] S104343. Use the multi-source copy model to combine the fusion result of the body copy vocabulary distribution weight and the body copy vocabulary distribution, and the fusion result of the title copy vocabulary distribution weight and the title copy vocabulary distribution to obtain the multi-source copy vocabulary distribution.

[0133] In an embodiment of the present invention, after the abstract acquisition device obtains the weight of the body copy vocabulary distribution and the weight of the title copy vocabulary distribution, it can use the multi-source copy model, the weight of the body copy vocabulary distribution, and the weight of the title copy vocabulary distribution to combine the body copy vocabulary distribution and the title copy vocabulary distribution. Here, the abstract acquisition device uses the multi-source copy model to combine the fusion result of the body copy vocabulary distribution weight and the body copy vocabulary distribution, and the fusion result of the title copy vocabulary distribution weight and the title copy vocabulary distribution to obtain the multi-source copy vocabulary distribution.

[0134] Further, in the embodiments of the present invention, the word processing model further includes a distribution weight determination model and a distribution combination model; wherein, the distribution weight determination model is used to determine the weights of the multi-source copy word list distribution and the generated word list distribution respectively when combining the multi-source copy word list distribution and the generated word list distribution; the distribution combination model is used to combine the multi-source copy word list distribution and the generated word list distribution based on the weights of the multi-source copy word list distribution and the generated word list distribution respectively. Thus, in S104, the abstract acquisition device uses the word processing model to combine the multi-source copy word list distribution and the generated word list distribution, including S1045-S1046, which will be described below in conjunction with each step.

[0135] S1045. Use the distribution weight determination model to perform a second non-linear combination on the current decoding feature, the text context vector, the title context vector, and the current decoding input feature to obtain the generated word list distribution weight; and obtain the opposite value of the generated word list distribution weight to obtain the copy word list distribution weight.

[0136] In the embodiments of the present invention, since the distribution weight determination model determines the weights of the multi-source copy word list distribution and the generated word list distribution respectively based on the current decoding feature, the text context vector, the title context vector, and the current decoding input feature when combining the multi-source copy word list distribution and the generated word list distribution; therefore, after the abstract acquisition device obtains the current decoding feature, the text context vector, the title context vector, and the current decoding input feature, it can use the distribution weight determination model to perform a second non-linear combination on the current decoding feature, the text context vector, the title context vector, and the current decoding input feature to obtain the generated word list distribution weight and the copy word list distribution weight; wherein, the result of the second non-linear combination is the generated word list distribution weight, and the opposite value of the generated word list distribution weight is the copy word list distribution weight.

[0137] Exemplarily, the distribution weight determination model is shown in Equation (6):

[0138]

[0139] where, P gen is the generated word list distribution weight, σ() is an existing function operation, and b2 are both training parameters in the distribution weight determination model, is the text context vector, is the title context vector, s t is the decoding feature at the t-th (current decoding time) moment (current decoding feature), x t is the input feature at the t-th (current decoding time) moment (current decoding input feature). Here, it is easy to know that the copy word list distribution weight is 1 - P gen .

[0140] S1046. Using the distribution combination model, combine the fusion result of the replicated vocabulary distribution weights and the multi-source replicated vocabulary distribution, and the fusion result of the generated vocabulary distribution weights and the generated vocabulary distribution, to complete the combination of the multi-source replicated vocabulary distribution and the generated vocabulary distribution.

[0141] In the embodiment of the present invention, after the abstract acquisition device obtains the generated vocabulary distribution weights, the replicated vocabulary distribution weights, the generated vocabulary distribution, and the replicated vocabulary distribution, it can use the distribution combination model to combine the fusion result of the replicated vocabulary distribution weights and the multi-source replicated vocabulary distribution, and the fusion result of the generated vocabulary distribution weights and the generated vocabulary distribution, to complete the combination of the multi-source replicated vocabulary distribution and the generated vocabulary distribution.

[0142] It should be noted that referring to Figure 5 , Figure 5 is a schematic structural diagram of the preset abstract acquisition model provided by the embodiment of the present invention. As Figure 5 shown, the preset abstract acquisition model 5-1 includes a text encoder 5-11, a title encoder 5-12, a decoder 5-13, and a word processing model 5-14; among them, the word processing model 5-14 includes a contribution model 5-141, a context vector model 5-142, a word replication model 5-143, a word generation model 5-144, a distribution weight determination model 5-145, and a distribution combination model 5-146; and, the word replication model 5-143 includes a word determination model 5-1431, a text word distribution model 5-1432, a title word distribution model 5-1433, and a replication distribution combination model 5-1434, and the replication distribution combination model 5-1434 includes a decoding input acquisition model 5-14341, a replication weight determination model 5-14342, and a multi-source replication model 5-14343. Thus, when the abstract acquisition device executes the abstract acquisition method provided by the embodiment of the present invention, first, it passes through the text encoder 5-11, the title encoder 5-12, and the decoder 5-13; then, it passes through the contribution model 5-141, the context vector model 5-142, the word determination model 5-1431, the text word distribution model 5-1432, the title word distribution model 5-1433, the decoding input acquisition model 5-14341, the replication weight determination model 5-14342, and the multi-source replication model 5-14343 in sequence; finally, it passes through the word generation model 5-144, the distribution weight determination model 5-145, and the distribution combination model 5-146 in sequence.

[0143] Further, referring to Figure 6 , in the embodiment of the present invention, before the abstract acquisition device obtains the preset abstract acquisition model in S101, the abstract acquisition method further includes S105-S110, which will be described below in combination with Figure 6 the steps shown.

[0144] S105. Obtain training samples and the original abstract acquisition model; the training samples are sample pairs composed of text samples, title samples, and abstract samples, and the original abstract acquisition model includes an original text encoder, an original title encoder, an original decoder, and an original word processing model.

[0145] In an embodiment of the present invention, the preset abstract acquisition model in the abstract acquisition device is pre-trained. Before the abstract acquisition device trains to obtain the preset abstract acquisition model, it is necessary to first obtain samples for training to obtain the preset abstract acquisition model, that is, obtain training samples; here, the training samples are sample pairs composed of text samples, title samples, and abstract samples, where the text samples and title samples are multimedia text samples, and the abstract samples are annotation data of the abstracts of the multimedia text samples.

[0146] In addition, the abstract acquisition device also needs to build an original abstract acquisition model and obtain the preset abstract acquisition model through continuous training of the original abstract acquisition model; here, the original abstract acquisition model is a model to be trained, and the original abstract acquisition model includes multiple sub-model structures to be trained: an original text encoder, an original title encoder, an original decoder, and an original word processing model; among them, the original text encoder is a model structure to be trained of the text encoder, used to extract the semantic features of the text samples; the original title encoder is a model structure to be trained of the title encoder, used to extract the semantic features of the title samples; the original decoder is a model structure to be trained of the decoder, used to decode the extracted semantic features of the text samples and title samples to obtain another information representation of the multimedia text samples; the original word processing model is a model structure to be trained of the word processing model, used to process another information representation of the multimedia text samples to determine the predicted abstract of the multimedia text samples.

[0147] It should be noted that the abstract acquisition device can also perform data augmentation on the training samples to improve the accuracy of the finally obtained preset abstract acquisition model.

[0148] S106. Use the original text encoder to encode the text samples to obtain text sample encoding features; and use the original title encoder to encode the title samples to obtain title sample encoding features.

[0149] In an embodiment of the present invention, after an abstract acquisition device obtains training samples including a text sample, a title sample, and an abstract sample, and an original abstract acquisition model including an original text encoder, an original title encoder, an original decoder, and an original word processing model, the original abstract acquisition model can be trained using the training samples. First, the abstract acquisition device uses the original text encoder to extract the semantic features of the text sample, that is, encodes the text sample to obtain a semantic feature representation of the text sample, and thus obtains the encoded feature of the text sample; and uses the original title encoder to extract the semantic features of the title sample, that is, encodes the title sample to obtain a semantic feature representation of the title sample, and thus obtains the encoded feature of the title sample.

[0150] S107. Use the original decoder to decode the encoded feature of the text sample and the encoded feature of the title sample at the current decoding training moment to obtain the current decoded sample feature; the current decoding training moment is any moment of the decoding training moment.

[0151] In an embodiment of the present invention, after the abstract acquisition device obtains the encoded feature of the text sample and the encoded feature of the title sample, since the encoded feature of the text sample represents the semantic features of the text sample and the encoded feature of the title sample represents the semantic features of the title sample; therefore, the abstract acquisition device can use the original decoder to represent the multimedia text sample as another information representation at each decoding training moment for the encoded feature of the text sample and the encoded feature of the title sample; here, for any moment of the decoding training moment, the other information representation of the decoded multimedia text sample is the current decoded sample feature.

[0152] S108. Use the original word processing model to combine the encoded feature of the text sample, the encoded feature of the title sample, and the current decoded sample feature to obtain a multi-source copy vocabulary sample distribution and a generated vocabulary sample distribution; and combine the multi-source copy vocabulary sample distribution and the generated vocabulary sample distribution to obtain a target vocabulary sample distribution.

[0153] In an embodiment of the present invention, the encoded feature of the text sample, the encoded feature of the title sample, and the current decoded sample feature are used to determine the distribution probability of the keywords corresponding to the current decoding training moment; therefore, after the abstract acquisition device obtains the encoded feature of the text sample, the encoded feature of the title sample, and the current decoded sample feature, it can use the original word processing model to process the encoded feature of the text sample, the encoded feature of the title sample, and the current decoded sample feature to determine the distribution probability of the keywords corresponding to the current decoding training moment.

[0154] Here, the abstract acquisition device combines the text encoding sample features, title encoding sample features, and current decoding sample features using a word processing model, and thus can obtain the distribution information of keywords in the multimedia text sample, that is, the multi-source copy vocabulary sample distribution; and can obtain the distribution information of keywords generated based on the multimedia text sample, that is, the generated vocabulary sample distribution, which also refers to the probability value of each keyword in the preset vocabulary corresponding to the output at the current decoding training moment.

[0155] Since the multi-source copy vocabulary distribution represents the probability distribution that the keywords output at the current decoding moment originate from the multimedia text, and the generated vocabulary distribution represents the probability distribution that the keywords output at the current decoding moment are generated words; therefore, the abstract acquisition device combines the multi-source copy vocabulary distribution and the generated vocabulary distribution using a word processing model, and the obtained target vocabulary sample distribution also includes the keywords output at the current decoding training moment.

[0156] S109. Obtain the difference between the target vocabulary sample distribution and the abstract sample to obtain the current loss function, so that when the decoding training moment ends, a target loss function composed of multiple current loss functions is obtained.

[0157] In the embodiment of the present invention, after the abstract acquisition device obtains the target vocabulary sample distribution at the current decoding training moment, since the target vocabulary sample distribution represents the current abstract acquisition ability of the original abstract acquisition sample, and the abstract sample is the real sample of the multimedia text sample; therefore, by obtaining the difference between the target vocabulary sample distribution and the abstract sample, the loss function at the current decoding training moment, that is, the current loss function, is obtained; at this time, the decoding at the current decoding training moment is completed. And when all decoding training moments end, multiple current loss functions are obtained, and the target loss function is obtained; at this time, one decoding is completed.

[0158] It should be noted that the abstract acquisition device can also process the probability value of the abstract sample in the target vocabulary sample distribution according to the maximum likelihood estimation method to obtain the current loss function.

[0159] S110. Continuously train the preset abstract acquisition model using the target loss function until the training cut-off condition is met, and obtain a preset abstract acquisition model including a text encoder, a title encoder, a decoder, and a word processing model.

[0160] In an embodiment of the present invention, since the target loss function characterizes the current decoding effect to a certain extent; therefore, after the abstract acquisition device obtains the target loss function, it can continuously adjust the parameters in the preset abstract acquisition model by using the target loss function to continuously train the preset abstract acquisition model; until it is determined that the training cut-off condition is met during the training process, the training ends, and the model at this time is the preset abstract acquisition model including the text encoder, the title encoder, the decoder, and the word processing model.

[0161] It should be noted that the process of continuously training the preset abstract acquisition model by using the target loss function is an iterative process. In addition, the text encoder is the original trained text encoder, the title encoder is the original trained title encoder, the decoder is the original trained encoder, and the word processing model is the original trained word processing model.

[0162] In an embodiment of the present invention, the abstract acquisition device also has the function of evaluating the preset abstract acquisition model.

[0163] Exemplarily, when the abstract acquisition device evaluates the results of the preset abstract acquisition model, it selects two metrics, ROUGE (including three metrics: ROUGE-1, ROUGE-2, and ROUGE-L) and human evaluation, for evaluation; among them, the preset abstract acquisition model 14-1 is the MSPG model, the evaluation comparison model 14-2 is the seq2seq and Attention mechanism (basic), the evaluation comparison model 14-3 is the pointer generation model, and the evaluation comparison model 14-4 is the pointer generation model and the coverage model; the evaluation results are as follows: the values corresponding to the evaluation indexes of the preset abstract acquisition model 14-1 are 39.19, 15.33, 30.12, and 55.6% in sequence, the values corresponding to the evaluation indexes of the evaluation comparison model 14-2 are 24.17, 8.32, 23.26, and 18.2% in sequence, the values corresponding to the evaluation indexes of the evaluation comparison model 14-3 are 32.7, 11.83, 25.63, and 33.3% in sequence, and the values corresponding to the evaluation indexes of the evaluation comparison model 14-4 are 35.22, 13.1, 27.87, and 39.7% in sequence.

[0164] Further, in an embodiment of the present invention, after the abstract acquisition device obtains the preset abstract acquisition model including the text encoder, the title encoder, the decoder, and the word processing model, the abstract acquisition method further includes S111-S112, which will be described below in combination with these steps.

[0165] S111. Obtain a new training sample; the new training sample is used to optimize the preset abstract acquisition model.

[0166] It should be noted that when the abstract acquisition device obtains the preset abstract acquisition model, it can also optimize the preset abstract acquisition model by obtaining new training samples to improve the generalization ability of the preset abstract acquisition model.

[0167] S112. Optimize the preset abstract acquisition model based on the new training samples to obtain an optimized preset abstract acquisition model.

[0168] In the embodiment of the present invention, after the abstract acquisition device obtains new training samples, it can optimize the preset abstract acquisition model based on the new training samples to use the optimized preset abstract acquisition model for abstract acquisition. Here, the optimization process is similar to the process of training the preset abstract acquisition model, and the embodiment of the present invention will not elaborate here.

[0169] Correspondingly, in the embodiment of the present invention, the abstract acquisition device uses the optimized preset abstract acquisition model to perform abstract acquisition on the multimedia text to obtain the target abstract.

[0170] It can be understood that by continuously optimizing the preset abstract acquisition model, the abstract acquisition device can improve the generalization ability of the optimized preset abstract acquisition model. Furthermore, when implementing abstract acquisition based on the optimized preset abstract acquisition model, the effect of abstract acquisition can be further improved.

[0171] Furthermore, in the embodiment of the present invention, the abstract acquisition device obtains the multimedia text in S101, including: the abstract acquisition device obtains the multimedia text from the preset task queue; the preset task queue is a queue composed of abstract acquisition tasks requested by the multimedia server.

[0172] Correspondingly, when the decoding moment ends and the target abstract composed of multiple current keywords is obtained, the abstract acquisition method further includes S113: the abstract acquisition device stores the target abstract and the multimedia text in the preset abstract cache, so that when the multimedia server obtains an information acquisition request sent by the terminal, it obtains the target abstract and the multimedia text from the preset abstract cache, and sends the target abstract and the multimedia text to the terminal to display the multimedia title in the first display area of the display interface of the terminal and display the target abstract in the second display area of the display interface of the terminal.

[0173] Here, when the terminal sends an information acquisition request to the multimedia server, the multimedia server responds to the information acquisition request, sends the multimedia text to a preset task queue, so that the summary acquisition device acquires the multimedia text from the preset task queue, and uses the summary acquisition method provided by the embodiment of the present invention for the multimedia text to acquire the target summary, and stores the target summary and the corresponding multimedia text in a preset summary cache; so that the multimedia server acquires the target summary and the corresponding multimedia text from the preset summary cache and returns them to the terminal for display; or the summary acquisition device stores the target summary and the multimedia text in a preset summary cache, so that the multimedia server acquires the target summary and the multimedia text from the preset summary cache, and when receiving the information acquisition request sent by the terminal, sends the target summary and the multimedia text to the terminal, so as to display the multimedia title in the first display area of the display interface of the terminal, and display the target summary in the second display area of the display interface of the terminal; the embodiment of the present invention does not make specific limitations on this.

[0174] In addition, the number of target summaries and multimedia texts is at least one respectively. At this time, at least one summary in the target summaries corresponds to at least one text in the multimedia texts one by one; when displayed on the display interface, the area for displaying the title of one text is followed by the area for displaying the summary in the corresponding one text.

[0175] It should be noted that when the multimedia server displays the target summary and the multimedia text, it is through the terminal for display. Here, when the terminal receives the summary display instruction, it sends an information acquisition request to the multimedia server to acquire the multimedia text and the target summary; receives the multimedia text and the target summary sent by the multimedia server in response to the information acquisition request, where the multimedia text includes the multimedia body and the multimedia title; displays the target summary in the first display area of the display interface, displays the multimedia title in the second display area of the display interface, and displays the relevant information (picture or abbreviated form of the multimedia body) of the multimedia body in the third display area of the display interface; when receiving an operation on any one of the first display area, the second display area, and the third display area, it jumps to the detailed browsing interface of the multimedia text, and the multimedia body is displayed in the detailed browsing interface of the multimedia text. In this way, when the user browses multimedia information through the display interface, the user can quickly and accurately determine the important information of the multimedia body through the multimedia title and the target summary, thereby improving operation behaviors such as the click-through rate of the multimedia body.

[0176] That is to say, the summary acquisition method in the embodiment of the present invention is implemented through a summary acquisition architecture as Figure 7 shown, such as Figure 7As shown, the abstract acquisition architecture includes a data source layer 7-1, a task receiving layer 7-2, an abstract acquisition layer 7-3, and a task distribution layer 7-4. Among them, the data source layer 7-1 is used to obtain the abstract acquisition tasks requested by the multimedia server and store them in the task pool. The task receiving layer 7-2 is used to obtain the abstract acquisition tasks from the task pool in the data source layer 7-1, put the obtained abstract acquisition tasks into a preset task queue (for example, Kafka queue) to wait for abstract acquisition processing. The abstract acquisition layer 7-3 is used to obtain the abstract acquisition tasks from the preset task queue. The abstract acquisition tasks include multimedia texts, and use a preset abstract acquisition model to perform abstract acquisition on the multimedia texts to obtain a target abstract, and store the corresponding relationship between the target abstract and the multimedia text in a preset abstract cache (for example, Redis cache). The task distribution layer 7-4 is used to distribute the multimedia texts and target abstracts in the preset abstract cache to the multimedia server to complete task feedback, so that the multimedia server can complete the display of the target abstract and the multimedia text.

[0177] Further, referring to Figure 8 , Figure 8 is another optional process schematic diagram of the abstract acquisition method provided by the embodiment of the present invention. As Figure 8 shown, after S104 in the embodiment of the present invention, S114 is further included. That is to say, when the decoding time ends and the abstract acquisition device obtains the target abstract composed of multiple current keywords, the abstract acquisition method further includes S114: sending the target abstract and the multimedia text to the blockchain network, so that the nodes of the blockchain network fill the target abstract and the multimedia text into a new block, and when the new block is consensus-consistent, append the new block to the tail of the blockchain to complete the chain-up. Here, the abstract acquisition device chains up the obtained target abstract and multimedia text, so as to ensure non-tampering.

[0178] Based on Figure 8 the abstract acquisition method shown, referring to Figure 9 , Figure 9 is another optional architecture schematic diagram of the abstract acquisition system 100 provided by the embodiment of the present invention, including a blockchain network 600 (exemplarily showing consensus nodes 610-1 to consensus nodes 610-3), an authentication center 700, a business entity 800, and a business entity 900, which will be described separately below.

[0179] The type of the blockchain network 600 is flexible and diverse. For example, it can be any one of a public blockchain, a private blockchain, or a consortium blockchain. Taking the public blockchain as an example, the electronic devices of any business entity, such as user terminals and servers, can access the blockchain network 600 without authorization. Taking the consortium blockchain as an example, after obtaining authorization, the electronic devices (such as terminals / servers) under the jurisdiction of a business entity can access the blockchain network 600. At this time, they become client nodes in the blockchain network 600.

[0180] In some embodiments, the client node can only be an observer of the blockchain network 600, that is, it provides functions to support a business entity in initiating transactions (for example, for storing data on the chain or querying data on the chain). For the functions of the nodes in the blockchain network 600, such as sorting functions, consensus services, and ledger functions, etc., the client node can be default or selectively (for example, depending on the specific business needs of the business entity) implemented. Thus, the data and business processing logic of the business entity can be migrated to the blockchain network 600 to the greatest extent, and the credibility and traceability of the data and business processing process can be achieved through the blockchain network 600.

[0181] The nodes in the blockchain network 600 receive transactions submitted by client nodes from different business entities (such as Figure 9 the business entity 800 shown in Figure 9 ), execute the transactions to update the ledger or query the ledger, and various intermediate results or final results of the executed transactions can be returned to the client nodes of the business entity for display.

[0182] Taking the access of multiple business entities to the blockchain network to implement the management of multimedia texts and target summaries as an example, the exemplary application of the blockchain network is described below.

[0183] Continue to refer to Figure 9 , the business entity 800 involved in the management link can be an abstract acquisition system based on artificial intelligence, and the business entity 900 can be a display system of multimedia texts. They obtain their respective digital certificates from the certification center 700. The digital certificates include the public keys of the business entities and the digital signatures signed by the certification center 700 for the public keys and identity information of the business entities. They are used to be attached to the transactions together with the digital signatures of the business entities for the transactions and sent to the blockchain network, so that the blockchain network can take out the digital certificates and signatures from the transactions, verify the reliability of the messages (that is, whether they have been tampered with) and the identity information of the business entities sending the messages. The blockchain network will verify according to the identity, such as whether it has the permission to initiate transactions. The clients running on the electronic devices (such as terminals or servers) under the jurisdiction of the business entity can all request access to the blockchain network 600 and become client nodes.

[0184] The client node 810 of the business entity 800 is used to obtain multimedia text; obtain the summary of the multimedia text by using a preset summary obtaining model to obtain a target summary; and send the multimedia text and the corresponding target summary to the blockchain network 600.

[0185] Among them, for the operation of sending the multimedia text and the corresponding target summary to the blockchain network 600, the business logic can be preset in the client node 810 in advance. When the target summary is generated, the client node 810 automatically sends the multimedia text and the corresponding target summary to the blockchain network 600. Or, the business personnel of the business entity 800 can log in to the client node 810, manually package the multimedia text and the target summary, and send them to the blockchain network 600. When sending, the client node 810 generates a transaction corresponding to the update operation according to the multimedia text and the target summary, specifies the smart contract to be called to implement the update operation and the parameters passed to the smart contract in the transaction. The transaction also carries the digital certificate of the client node 810 and the signed digital signature (for example, encrypted by using the private key in the digital certificate of the client node 810 for the summary of the transaction), and broadcasts the transaction to the consensus nodes in the blockchain network 600.

[0186] When the consensus node in the blockchain network 600 receives the transaction, it verifies the digital certificate and digital signature carried by the transaction. After successful verification, it confirms whether the business entity 800 has the transaction permission according to the identity of the business entity 800 carried in the transaction. Any verification judgment in the digital signature and permission verification will cause the transaction to fail. After successful verification, it signs its own digital signature (for example, encrypted by using the private key of the consensus node 610-1 for the summary of the transaction), and continues to broadcast in the blockchain network 600.

[0187] After the consensus node in the blockchain network 600 receives the transaction with successful verification, it fills the transaction into a new block and broadcasts it. When the consensus node in the blockchain network 600 broadcasts the new block, it conducts a consensus process on the new block. If the consensus is successful, it appends the new block to the tail of the blockchain stored by itself, and updates the state database according to the result of the transaction, and executes the transaction in the new block: for the transaction of submitting the update of the multimedia text and the target summary, add a key-value pair including the multimedia text and the target summary to the state database.

[0188] The business personnel of the business entity 900 log in to the client node 910 (terminal 400), input a summary acquisition query request, and the client node 910 generates a transaction corresponding to the update operation / query operation according to the summary acquisition query request. The smart contract to be called for implementing the update operation / query operation and the parameters to be passed to the smart contract are specified in the transaction. The transaction also carries the digital certificate of the client node 910 and the signed digital signature (for example, encrypted using the private key in the digital certificate of the client node 910 for the digest of the transaction), and broadcasts the transaction to the consensus nodes in the blockchain network 600. Among them, the summary acquisition query request is used to query the multimedia text and the corresponding target summary.

[0189] After receiving the transaction, the consensus nodes in the blockchain network 600 verify the transaction, fill the block, and reach a consensus. Then, the filled new block is appended to the tail of the blockchain stored by itself, and the status database is updated according to the result of the transaction, and the transaction in the new block is executed; for example, for the transaction of submitting a query for a certain multimedia text, query the key-value pair corresponding to the multimedia text from the status database and return the transaction result.

[0190] It should be noted that in Figure 9 an exemplary process of directly uploading the multimedia text and the corresponding target summary to the blockchain is shown, but in some other embodiments, for the case where the data volume of the multimedia text is large, the client node 810 can upload the hash of the multimedia text and the hash of the target summary in pairs, and store the original multimedia text and the target summary in a distributed file system or database. After the client node 910 obtains the multimedia text and the target summary from the distributed file system or database, it can perform verification in combination with the corresponding hash in the blockchain network 600, thereby reducing the workload of the uploading operation.

[0191] As an example of the blockchain, see Figure 10 , Figure 10 is a schematic structural diagram of the blockchain in the blockchain network 600 provided by the embodiments of the present invention. The header of each block can include both the hash values of all transactions in the block and the hash values of all transactions in the previous block. After the record of the newly generated transaction is filled into the block and consensus is reached by the nodes in the blockchain network, it will be appended to the tail of the blockchain to form a chain-like growth. The chain-like structure based on the hash values between blocks ensures the anti-tampering and anti-forgery of the transactions in the block.

[0192] The following describes the exemplary functional architecture of the blockchain network provided by the embodiments of the present invention. See Figure 11 , Figure 11It is a schematic diagram of the functional architecture of the blockchain network 600 provided by an embodiment of the present invention, including an application layer 601, a consensus layer 602, a network layer 603, a data layer 604, and a resource layer 605, which will be described separately below.

[0193] The resource layer 605 encapsulates the computing resources, storage resources, and communication resources of each consensus node 610 in the blockchain network 600.

[0194] The data layer 604 encapsulates various data structures for implementing the ledger, including the blockchain implemented by files in the file system, the key-value state database, and the proof of existence (such as the hash tree of transactions in a block).

[0195] The network layer 603 encapsulates the functions of the peer-to-peer (P2P) network protocol, data dissemination mechanism, data verification mechanism, access authentication mechanism, and business entity identity management.

[0196] Among them, the P2P network protocol realizes the communication between nodes in the blockchain network 600, the data dissemination mechanism ensures the dissemination of transactions in the blockchain network 60, and the data verification mechanism is used to realize the reliability of data transmitted between nodes based on cryptographic methods (such as digital certificates, digital signatures, public / private key pairs); the access authentication mechanism is used to authenticate the identity of business entities joining the blockchain network 600 according to the actual business scenario, and grant the business entity the permission to access the blockchain network 600 when the authentication is passed; the business entity identity management is used to store the identities of business entities allowed to access the blockchain network 600 and their permissions (such as the types of transactions that can be initiated).

[0197] The consensus layer 602 encapsulates the mechanism for the consensus nodes in the blockchain network 600 to reach consensus on blocks (i.e., the consensus mechanism), the functions of transaction management and ledger management. The consensus mechanism includes consensus algorithms such as POS, POW, and DPOS, and supports the pluggability of consensus algorithms.

[0198] Transaction management is used to verify the digital signatures carried in the transactions received by the nodes, verify the identity information of business entities, and determine whether they have the permission to conduct transactions based on the identity information (read relevant information from the business entity identity management); for business entities authorized to access the blockchain network 600, they all have digital certificates issued by the certification center. The business entity uses the private key in its digital certificate to sign the submitted transaction, thereby declaring its legal identity.

[0199] Ledger management is used to maintain the blockchain and the state database. For the blocks that have reached consensus, append them to the end of the blockchain; execute the transactions in the blocks that have reached consensus. When a transaction includes an update operation, update the key-value pairs in the state database. When a transaction includes a query operation, query the key-value pairs in the state database and return the query results to the client nodes of the business entity. Support multiple dimensions of query operations on the state database, including: querying a block according to the block sequence number (such as the hash value of a transaction); querying a block according to the block hash value; querying a block according to the transaction sequence number; querying a transaction according to the transaction sequence number; querying the account data of a business entity according to the account (sequence number) of the business entity; querying the blockchain in a channel according to the channel name.

[0200] The application layer 601 encapsulates various services that the blockchain network can implement, including the traceability, storage, and verification of transactions, etc.

[0201] Next, the exemplary application of the embodiments of the present invention in a practical application scenario will be described.

[0202] Exemplarily, refer to Figure 12 , Figure 12 is a schematic diagram of the application of an exemplary abstract acquisition method provided by the embodiments of the present invention; as Figure 12As shown, the news information includes news text 12-1 and news title 12-2. First, the news text 12-1 (multimedia text) is input into Encoder1 12-3 (text encoder) for encoding to obtain an encoding state 12-4 (text encoding feature), and the news title 12-2 (multimedia title) is input into Encoder2 12-5 (title encoder) for encoding to obtain an encoding state 12-6 (title encoding feature). Second, Decoder 12-7 decodes at the current decoding moment based on the encoding state 12-4 and the encoding state 12-6 to obtain an output state 12-8 (current decoding feature). Then, based on the encoding state 12-4 and the output state 12-8, the Attention value 12-9 (text information contribution degree) corresponding to the news text 12-1 is obtained, and then the context vector 12-10 (text context vector) is obtained; based on the encoding state 12-6 and the output state 12-8, the Attention value 12-11 (title information contribution degree) corresponding to the news title 12-5 is obtained, and then the context vector 12-12 (title context vector) is obtained. Next, based on the context vector 12-10, the context vector 12-12, and the output state 12-8, the copy weight λ12-13 is determined, and then based on the copy weight λ12-13, the Attention value 12-9, and the Attention value 12-11, the multi-source copy vocabulary distribution 12-14 is obtained; and based on the context vector 12-10, the context vector 12-12, and the output state 12-8, the distribution weight 12-15 is determined, and at the same time, based on the context vector 12-10, the context vector 12-12, and the output state 12-8, the generated vocabulary distribution 12-16 is determined. Then, based on the distribution weight 12-15, the multi-source copy vocabulary distribution 12-14, and the generated vocabulary distribution 12-16, the current keyword 12-17 is determined, and then the abstract 12-18 (target abstract) is obtained.

[0203] Exemplarily, refer to Figure 13 , Figure 13 is a schematic diagram showing an exemplary target abstract provided by an embodiment of the present invention; as Figure 13 shown, in the recommended column of the display interface 13-1 of the terminal in the embodiment of the present invention, the display area 13-12 (first display area) displays the news title (multimedia title), the display area 13-13 (second display area) displays the news abstract (target abstract), and the display area 13-11 (third display area) displays the picture of the news text (multimedia text); thus, by using the abstract acquisition method provided by the embodiment of the present invention, it can be realized that the click-through rate of news information is improved through high-quality target abstracts. Among them, the multimedia server provides service functions for the terminal.

[0204] Next, the exemplary structure of the abstract acquisition device 255 provided in the embodiments of the present invention implemented as a software module will be further described. In some embodiments, as Figure 3 shown, the software module stored in the abstract acquisition device 255 in the memory 250 may include:

[0205] An acquisition module 2551, configured to acquire a multimedia text and acquire a preset abstract acquisition model; the multimedia text includes a multimedia body and a multimedia title, and the preset abstract acquisition model includes a body encoder, a title encoder, a decoder, and a word processing model;

[0206] An encoding module 2552, configured to use the body encoder to encode the multimedia body to obtain a body encoding feature; and use the title encoder to encode the multimedia title to obtain a title encoding feature;

[0207] A decoding module 2553, configured to use the decoder to decode the body encoding feature and the title encoding feature at the current decoding time to obtain a current decoding feature; the current decoding time is any time of the decoding time;

[0208] An abstract acquisition module 2554, configured to use the word processing model to combine the body encoding feature, the title encoding feature, and the current decoding feature to obtain a multi-source copy word table distribution and a generated word table distribution; and combine the multi-source copy word table distribution and the generated word table distribution, and determine a current keyword according to the combination result, so as to obtain a target abstract composed of multiple current keywords when the decoding time ends; wherein, the multi-source copy word table distribution is the distribution information of keywords in the multimedia text, and the generated word table distribution is the distribution information of keywords generated based on the multimedia text.

[0209] Further, the word processing model includes a contribution model, a context vector model, a word replication model, and a word generation model; the summary acquisition module 2554 is further configured to use the contribution model to perform a first normalization process on the combination of the body text encoding feature and the current decoding feature, and the combination of the title encoding feature and the current decoding feature, respectively, to obtain the body text information contribution corresponding to the body text encoding feature, and the title information contribution corresponding to the title encoding feature; use the context vector model to fuse the body text encoding feature with the body text information contribution to obtain a body text context vector; and fuse the title encoding feature with the title information contribution to obtain a title context vector; use the word replication model to fuse the body text information contribution and the title information contribution based on the current decoding feature to obtain the multi-source replication word table distribution; use the word generation model to perform a second normalization process on the body text context vector, the title context vector, and the current decoding feature to obtain the generated word table distribution.

[0210] Further, the word replication model includes a word determination model, a body text word distribution model, a title word distribution model, and a replication distribution combination model; the summary acquisition module 2554 is further configured to use the word determination model to copy keywords from the multimedia body text and the multimedia title, respectively, to obtain a body text replication word table corresponding to the multimedia body text, and a title replication word table corresponding to the multimedia title; use the body text word distribution model to determine a target body text word table that matches the current decoding feature from the body text replication word table, and determine a target body text information contribution corresponding to each keyword of the target body text word table from the body text information contribution, and perform a weighted sum on the target body text information contribution to obtain the body text replication word table distribution corresponding to the body text replication word table; use the title word distribution model to determine a target title word table that matches the current decoding feature from the title replication word table, and determine a target title information contribution corresponding to each keyword of the target title word table from the title information contribution, and perform a weighted sum on the target title information contribution to obtain the title replication word table distribution corresponding to the title replication word table; use the replication distribution combination model to combine the body text replication word table distribution and the title replication word table distribution to obtain the multi-source replication word table distribution.

[0211] Further, the replication distribution combination model includes a decoding input acquisition model, a replication weight determination model, and a multi-source replication model; the summary acquisition module 2554 is further configured to use the decoding input acquisition model to obtain the current decoding input feature at the current decoding moment; use the replication weight determination model to perform a first non-linear combination on the body context vector, the title context vector, the current decoding state, and the current decoding input feature to obtain a body replication vocabulary distribution weight, and obtain an opposite value of the body replication vocabulary distribution weight to obtain a title replication vocabulary distribution weight; use the multi-source replication model to combine the fusion result of the body replication vocabulary distribution weight and the body replication vocabulary distribution, and the fusion result of the title replication vocabulary distribution weight and the title replication vocabulary distribution to obtain the multi-source replication vocabulary distribution.

[0212] Further, the word processing model further includes a distribution weight determination model and a distribution combination model; the summary acquisition module 2554 is further configured to use the distribution weight determination model to perform a second non-linear combination on the current decoding feature, the body context vector, the title context vector, and the current decoding input feature to obtain a generated vocabulary distribution weight; and obtain an opposite value of the generated vocabulary distribution weight to obtain a replication vocabulary distribution weight; use the distribution combination model to combine the fusion result of the replication vocabulary distribution weight and the multi-source replication vocabulary distribution, and the fusion result of the generated vocabulary distribution weight and the generated vocabulary distribution to complete the combination of the multi-source replication vocabulary distribution and the generated vocabulary distribution.

[0213] Further, the abstract acquisition device 255 further includes a model training module 2555, configured to obtain training samples and an original abstract acquisition model; the training samples are sample pairs composed of text samples, title samples, and abstract samples, and the original abstract acquisition model includes an original text encoder, an original title encoder, an original decoder, and an original word processing model; using the original text encoder, encoding the text samples to obtain text sample encoding features; and using the original title encoder, encoding the title samples to obtain title sample encoding features; using the original decoder, decoding the text sample encoding features and the title sample encoding features at the current decoding training moment to obtain current decoded sample features; the current decoding training moment is any moment of the decoding training moment; using the original word processing model, combining the text encoded sample features, the title encoded sample features, and the current decoded sample features to obtain a multi-source copy vocabulary sample distribution and a generated vocabulary sample distribution; and combining the multi-source copy vocabulary sample distribution and the generated vocabulary sample distribution to obtain a target vocabulary sample distribution; obtaining the difference between the target vocabulary sample distribution and the abstract samples to obtain a current loss function, so that when the decoding training moment ends, a target loss function composed of multiple current loss functions is obtained; continuously training the preset abstract acquisition model using the target loss function until the training cut-off condition is met, and obtaining the preset abstract acquisition model including the text encoder, the title encoder, the decoder, and the word processing model.

[0214] Further, the acquisition module 2551 is further configured to obtain the multimedia text from a preset task queue; the preset task queue is a queue composed of abstract acquisition tasks requested by a multimedia server;

[0215] Correspondingly, the abstract acquisition device 255 further includes a storage module 2556, configured to store the target abstract and the multimedia text in a preset abstract cache, so that

[0216] when the multimedia server receives an information acquisition request sent by a terminal, obtaining the target abstract and the multimedia text from the preset abstract cache, and sending the target abstract and the multimedia text to the terminal, so as to display the multimedia title in a first display area of the display interface of the terminal, and display the target abstract in a second display area of the display interface of the terminal.

[0217] Further, the abstract acquisition device 255 further includes a blockchain module 2557, configured to send the target abstract and the multimedia text to a blockchain network, so that nodes of the blockchain network fill the target abstract and the multimedia text into a new block, and when reaching a consensus on the new block, append the new block to the tail of the blockchain to complete the chain - up.

[0218] An embodiment of the present invention provides a computer storage medium storing executable instructions, where the executable instructions, when executed by a processor, cause the processor to execute the abstract acquisition method provided by the embodiment of the present invention. For example, Figure 4 the abstract acquisition method shown.

[0219] In some embodiments, the storage medium may be a FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD - ROM, etc.; or may be various devices including one or any combination of the above memories.

[0220] In some embodiments, the executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, sub - routine, or other unit suitable for use in a computing environment.

[0221] As an example, the executable instructions may or may not correspond to a file in the file system, may be stored as part of a file storing other programs or data. For example, stored in one or more scripts in a HyperText Markup Language (HTML) document, stored in a single file dedicated to the program under discussion, or stored in multiple cooperating files (such as files storing one or more modules, sub - routines, or code portions).

[0222] As an example, the executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected through a communication network.

[0223] In summary, through the embodiments of the present invention, when obtaining the target summary corresponding to the multimedia text, the model adopted determines the probability value of the decoded keyword from two aspects: the keyword in the multimedia text and the keyword generated based on the multimedia text. This makes the obtained target summary not only have the accuracy of the summary obtained by the summary extraction method but also have the rich semantics of the summary obtained by the summary generation method. In addition, both the text and the title of the multimedia text are used as the copy sources of the decoded keyword, implementing the technical solution of using the title of the multimedia text as the guiding information to extract keywords from the multimedia text as part of the information of the target summary. Since the title of the multimedia text can accurately express the important meaning of the multimedia text, the accuracy of the obtained target summary is further improved. Therefore, the accuracy and semantic richness of the obtained summary are improved.

[0224] As described above, the above are only embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and scope of the present invention are all included in the protection scope of the present invention.

Claims

1. A method for obtaining a summary, characterized in that: include: Acquire multimedia text and a preset summary acquisition model; the multimedia text includes a multimedia text and a multimedia title, and the preset summary acquisition model includes a text encoder, a title encoder, a decoder, and a word processing model; Using the text encoder, encoding the multimedia text to obtain a text encoding feature; and using the title encoder, encoding the multimedia title to obtain a title encoding feature; Using the decoder, decoding the text encoding feature and the title encoding feature at a current decoding moment to obtain a current decoding feature; the current decoding moment is any moment of the decoding moment; Using the word processing model, combining the text encoding features, the title encoding features, and the current decoding features to obtain a multi-source copy word list distribution and a generated word list distribution; combining the multi-source copy word list distribution and the generated word list distribution, and determining a current keyword based on the combination result, so as to obtain a target summary consisting of multiple current keywords when the decoding moment ends; The multi-source copied word list distribution is the distribution information of the keywords in the multimedia text, and the generated word list distribution is the distribution information of the keywords generated based on the multimedia text.

2. The method according to claim 1, characterized in that The word processing model includes a contribution model, a context vector model, a word replication model, and a word generation model; the word processing model is used to combine the text encoding features, the title encoding features, and the current decoding features to obtain a multi-source replication word list distribution and a generated word list distribution, including: Using the contribution model, performing a first normalization process on the combination of the text encoding feature and the current decoding feature, and the combination of the title encoding feature and the current decoding feature, respectively, to obtain a text information contribution corresponding to the text encoding feature, and a title information contribution corresponding to the title encoding feature; Using the context vector model, the text encoding feature is fused with the text information contribution to obtain a text context vector; and the title encoding feature is fused with the title information contribution to obtain a title context vector; Using the word replication model, based on the current decoding features, the text information contribution and the title information contribution are integrated to obtain the multi-source replication word list distribution; The word generation model is used to perform a second normalization process on the text context vector, the title context vector, and the current decoding feature to obtain the generated word list distribution.

3. The method according to claim 2, characterized in that The word replication model includes a word determination model, a text word distribution model, a title word distribution model, and a replication distribution combination model; the word replication model is used to fuse the text information contribution and the title information contribution based on the current decoding features to obtain the multi-source replication word list distribution, including: Using the word determination model, copy keywords from the multimedia text and the multimedia title respectively to obtain a text copy word list corresponding to the multimedia text and a title copy word list corresponding to the multimedia title; Using the text word distribution model, determining a target text word list that matches the current decoding feature from the text copy word list, determining a target text information contribution corresponding to each keyword in the target text word list from the text information contribution, performing a weighted summation on the target text information contribution to obtain the text copy word list distribution corresponding to the text copy word list; Using the title word distribution model, determining a target title word list that matches the current decoding feature from the title copy word list, determining a target title information contribution corresponding to each keyword in the target title word list from the title information contribution, performing a weighted summation on the target title information contribution to obtain the title copy word list distribution corresponding to the title copy word list; The multi-source replication vocabulary distribution is obtained by combining the body replication vocabulary distribution and the title replication vocabulary distribution using the replication distribution combination model.

4. The method according to claim 3, characterized in that The replication distribution combination model includes a decoding input acquisition model, a replication weight determination model, and a multi-source replication model; the multi-source replication vocabulary distribution is obtained by combining the text replication vocabulary distribution and the title replication vocabulary distribution using the replication distribution combination model, including: Obtaining a current decoding input feature at the current decoding moment using the decoding input acquisition model; Using the copy weight determination model, performing a first nonlinear combination on the text context vector, the title context vector, the current decoding state, and the current decoding input feature to obtain a text copy vocabulary distribution weight, and obtaining an opposite value of the text copy vocabulary distribution weight to obtain a title copy vocabulary distribution weight; Using the multi-source replication model, the fusion result of the text replication vocabulary distribution weight and the text replication vocabulary distribution, and the fusion result of the title replication vocabulary distribution weight and the title replication vocabulary distribution are combined to obtain the multi-source replication vocabulary distribution.

5. The method according to claim 4, characterized in that The word processing model further includes a distribution weight determination model and a distribution combination model; the use of the word processing model to combine the multi-source copied word list distribution and the generated word list distribution includes: Using the distribution weight determination model, performing a second nonlinear combination on the current decoding feature, the text context vector, the title context vector, and the current decoding input feature to obtain a generated vocabulary distribution weight; and obtaining an opposite value of the generated vocabulary distribution weight to obtain a copied vocabulary distribution weight; Utilizing the distribution combination model, the fusion result of the copied vocabulary distribution weight and the multi-source copied vocabulary distribution, as well as the fusion result of the generated vocabulary distribution weight and the generated vocabulary distribution are combined to complete the combination of the multi-source copied vocabulary distribution and the generated vocabulary distribution.

6. The method according to any one of claims 1 to 5, characterized in that Before obtaining the preset summary acquisition model, the method further includes: Obtaining training samples and an original summary acquisition model; the training samples are sample pairs consisting of a text sample, a title sample, and a summary sample; the original summary acquisition model includes an original text encoder, an original title encoder, an original decoder, and an original word processing model; Using the original text encoder, encoding the text sample to obtain a text sample encoding feature; and using the original title encoder, encoding the title sample to obtain a title sample encoding feature; Using the original decoder, decoding the text sample encoding features and the title sample encoding features at the current decoding training moment to obtain current decoding sample features; the current decoding training moment is any moment of the decoding training moment; Using the original word processing model, combining the text encoding sample features, the title encoding sample features, and the current decoding sample features to obtain a multi-source copied word list sample distribution and a generated word list sample distribution; and combining the multi-source copied word list sample distribution and the generated word list sample distribution to obtain a target word list sample distribution; Obtaining the difference between the target vocabulary sample distribution and the summary sample to obtain a current loss function, so as to obtain a target loss function composed of multiple current loss functions when the decoding training moment ends; The preset summary acquisition model is continuously trained using the target loss function until a training cutoff condition is met, thereby obtaining the preset summary acquisition model including the text encoder, the title encoder, the decoder, and the word processing model.

7. The method according to any one of claims 1 to 5, characterized in that The obtaining of multimedia text includes: Obtaining the multimedia text from a preset task queue; the preset task queue is a queue consisting of summary acquisition tasks requested by the multimedia server; Accordingly, when the decoding moment ends and a target summary consisting of multiple current keywords is obtained, the method further includes: The target summary and the multimedia text are stored in a preset summary cache so that When the multimedia server receives the information acquisition request sent by the terminal, it obtains the target summary and the multimedia text from the preset summary cache, and sends the target summary and the multimedia text to the terminal, so as to display the multimedia title in the first display area of the display interface of the terminal and display the target summary in the second display area of the terminal display interface.

8. The method according to any one of claims 1 to 5, characterized in that When the decoding moment ends and a target summary consisting of multiple current keywords is obtained, the method further includes: The target summary and the multimedia text are sent to the blockchain network so that The nodes of the blockchain network fill the target summary and the multimedia text into a new block, and when a consensus is reached on the new block, the new block is appended to the end of the blockchain to complete the chain.

9. A summary acquisition device, characterized in that: include: a memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 8 when executing the executable instructions stored in the memory.

10. A computer-readable storage medium, characterized in that Executable instructions are stored, which are used to cause a processor to execute and implement the method according to any one of claims 1 to 8.

Images