A method of abstract generation and related apparatus

Abstract

Embodiments of the present application disclose a method for generating an abstract and a related device, at least involving natural language processing, machine learning, computer vision and other technologies in artificial intelligence. The method comprises: performing feature extraction processing on text information in a to-be-processed document to obtain a text feature vector corresponding to the text information, and performing feature extraction processing on pictures in the to-be-processed document to obtain a picture feature vector corresponding to the pictures; performing self-attention processing on the text feature vector to obtain a first text feature vector; performing self-attention processing on the first text feature vector and the picture feature vector to obtain a target feature vector; and performing prediction processing on the target feature vector and a first abstract word based on an abstract generation model to generate target abstract information of the to-be-processed document. In the foregoing manner, the target abstract information of the to-be-processed document can be accurately generated, so that the target abstract information accurately expresses the main content required to be expressed by the document, and the accuracy of the abstract is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method and related apparatus for abstract generation. Background Technology

[0002] Currently, an increasing number of technologies are being applied to research in the field of text summarization. Automatic summarization is a technology that utilizes computers to automatically perform text analysis, content summarization, and automatic summary generation. It can express the main content of the original text in a concise form according to the requirements of the target audience. Automatic summarization technology can effectively help or enable users to find content of interest in retrieved articles, improving reading speed and quality. This automatic summarization technology can compress documents into more concise expressions and cover valuable topics in the original document.

[0003] In related solutions, a deep Transformer network architecture is typically used to process the text information and all related images in a document to generate a summary. Specifically, text features are extracted from the text information to obtain text features, and visual features are extracted from all images in the document to obtain corresponding visual features. Then, without changing the existing Transformer architecture, the text features and all visual features are simply concatenated and fused, and a decoder is used to decode the concatenated and fused feature vector to generate the corresponding summary information for the document. However, this solution needs to consider all images in the document, but does not pay attention to whether each image contributes to the generated summary. This results in the inclusion of many useless visual features from images in the subsequent feature fusion process, leading to the generated summary information being insufficient to accurately express the main content of the document and reducing the accuracy of the summary. Summary of the Invention

[0004] This application provides a method and related apparatus for generating summaries. It can accurately generate target summary information of a document without focusing on the image features of useless images, so that the target summary information accurately expresses the main content that the document needs to express, thereby improving the accuracy of the summary.

[0005] In a first aspect, embodiments of this application provide a method for generating summaries. The method includes: performing feature extraction processing on text information in a document to be processed to obtain a text feature vector corresponding to the text information; performing feature extraction processing on an image in the document to be processed to obtain an image feature vector corresponding to the image; performing self-attention processing on the text feature vector to obtain a first text feature vector; performing self-attention processing on the first text feature vector and the image feature vector to obtain a target feature vector; and performing prediction processing on the target feature vector and a first summary word based on a summarization generation model to generate target summary information for the document to be processed, wherein the first summary word is a historical summary word predicted from the target summary information of the document to be processed.

[0006] Secondly, embodiments of this application provide a summary generation apparatus. This summary generation apparatus includes a processing unit and an acquisition unit. The processing unit is used to perform feature extraction processing on text information in a document to be processed, obtaining a text feature vector corresponding to the text information, and to perform feature extraction processing on images in the document to be processed, obtaining an image feature vector corresponding to the image, where the image is used to indicate the illustration situation in the document to be processed; the processing unit is used to perform self-attention processing on the text feature vector to obtain a first text feature vector; the processing unit is used to perform self-attention processing on the first text feature vector and the image feature vector to obtain a target feature vector; the processing unit is used to perform prediction processing on the target feature vector and a first summary word based on a preset summary generation model to generate target summary information for the document to be processed, where the first summary word is a historical summary word predicted from the target summary information of the document to be processed.

[0007] In some optional examples, the processing unit is used to: fuse the first text feature vector with the image feature vector to obtain a fused feature vector; and perform self-attention processing on the fused feature vector and the first text feature vector to obtain a target feature vector.

[0008] In some alternative examples, the processing unit is used to: perform self-attention processing on the first text feature vector output by the (i-1)th encoding layer in the encoder to obtain the first text feature vector of the i-th encoding layer, where 1 < i ≤ L, L is a natural number, and L is the total number of encoding layers in the encoder; perform self-attention processing on the first text feature vector of the i-th encoding layer and the fused feature vector output by the (i-1)th encoding layer in the encoder to obtain the fused feature vector of the i-th encoding layer; and perform self-attention processing on the fused feature vector of the i-th encoding layer and the first text feature vector of the i-th encoding layer in the encoder to obtain the target feature vector.

[0009] In some alternative examples, the processing unit is used to: determine the query matrix in the image-aware self-attention mechanism based on the fused feature vector output by the i-th encoding layer and a preset first weight matrix; determine the key matrix in the image-aware self-attention mechanism based on the first text feature vector output by the i-th encoding layer and a preset second weight matrix; and determine the transpose matrix in the image-aware self-attention mechanism based on the first text feature vector output by the i-th encoding layer and a preset third weight matrix; and determine the target feature vector based on the query matrix, the key matrix, and the transpose matrix.

[0010] In some alternative examples, the processing unit is used to: calculate the similarity between the fused feature vector output by the i-th coding layer and the first text feature vector output by each i-th coding layer, based on the query matrix and the key matrix; and perform a weighted summation based on each similarity and the transpose matrix to determine the target feature vector.

[0011] In some alternative examples, the processing unit is used to: perform self-attention processing on the first summary words to obtain historical summary feature vectors; perform self-attention processing on the target feature vector and historical summary feature vectors to obtain a first feature vector; use the first feature vector as input to the summary generation model to obtain the predicted probability of each word; and generate target summary information of the document to be processed based on the predicted probability of each word.

[0012] In some alternative examples, the processing unit is also used to: sum the predicted probabilities of each word to obtain the target loss value; and update the parameters of the initial summary generation model based on the target loss value to obtain the summary generation model.

[0013] In some alternative examples, the acquisition unit is used to: acquire the word embedding feature representation corresponding to the first summary word. The processing unit is used to: perform self-attention processing on the word embedding feature representation corresponding to the first summary word to obtain the historical summary feature vector.

[0014] In some alternative examples, the acquisition unit is used to: acquire the semantic vector and position vector of each word in the text information of the document to be processed. The processing unit is used to: fuse the semantic vector and position vector of each word to obtain the text feature vector corresponding to the text information.

[0015] In some alternative examples, the acquisition unit is used to: perform feature extraction processing on the images in the document to be processed, and acquire the object feature vector, the identifier feature vector, the boundary feature vector of each object, and the identifier feature vector of the image for each object in the image. The processing unit is used to: concatenate the object feature vector, the identifier feature vector, the boundary feature vector, and the identifier feature vector of the image for each object to obtain the image feature vector corresponding to the image.

[0016] A third aspect of this application provides a digest generation apparatus, including: a memory, an input / output (I / O) interface, and a processor. The memory stores program instructions. The processor executes the program instructions in the memory to perform the digest generation method corresponding to the embodiment of the first aspect described above.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method corresponding to the embodiments of the first aspect described above.

[0018] The fifth aspect of this application provides a computer program product containing instructions that, when run on a computer or processor, causes the computer or processor to execute the method described above for performing the implementation method of the first aspect.

[0019] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0020] In this embodiment, feature extraction is performed on the text information in the document to be processed to obtain the text feature vector corresponding to the text information, and feature extraction is performed on the images in the document to obtain the image feature vector corresponding to the images. Then, self-attention processing is further applied to the text feature vector to obtain the first text feature vector, enabling the subsequent model to focus on the key content in the text information. Self-attention processing is also performed on the first text feature vector and the image feature vector to obtain the target feature vector, further enabling the subsequent model to focus on the key text and key images in the text information and images, thereby discarding the content in useless images. The target feature vector can be determined using the text feature vector of the key text and the image feature vector of the key images, ensuring that the obtained target feature vector does not include the image feature vector corresponding to useless images. Thus, based on the summarization generation model, the target feature vector and the first summary words are predicted, accurately generating the target summary information of the document to be processed, ensuring that the target summary information accurately expresses the main content required by the document, and improving the accuracy of the summary. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This diagram illustrates the architecture for generating summary information in existing schemes.

[0023] Figure 2 This illustration shows an application scenario diagram of the abstract generation method provided in the embodiments of this application;

[0024] Figure 3 This paper illustrates a schematic diagram of the system flow framework for abstract generation provided in an embodiment of this application;

[0025] Figure 4 A flowchart of a method for generating an abstract provided in an embodiment of this application is shown;

[0026] Figure 5 A schematic diagram illustrating the extraction of image feature vectors provided in an embodiment of this application is shown;

[0027] Figure 6 A schematic diagram of the processing flow of the image-aware self-attention mechanism provided in an embodiment of this application is shown;

[0028] Figure 7 This is a schematic diagram of one embodiment of the abstract generation apparatus provided in this application;

[0029] Figure 8 This is a schematic diagram of the hardware structure of the abstract generation device provided in the embodiments of this application. Detailed Implementation

[0030] This application provides a method and related apparatus for generating summaries. It can accurately generate target summary information of a document without focusing on the image features of useless images, so that the target summary information accurately expresses the main content that the document needs to express, thereby improving the accuracy of the summary.

[0031] It is understood that in the specific implementation of this application, user information, user personal data and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] The abstract generation method provided in this application is based on artificial intelligence (AI). Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making functions.

[0035] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0036] In the embodiments of this application, the main artificial intelligence technologies involved include the aforementioned natural language processing (NLP), machine learning (ML), and computer vision. For example, it may involve text preprocessing in natural language processing; it may also involve deep learning in machine learning, including self-attention processing and embedding. It may also involve image processing and image semantic understanding in computer vision, but the embodiments of this application do not provide specific limitations.

[0037] The digest generation method provided in this application can be applied to digest generation devices with data processing capabilities, such as terminal devices and servers. Terminal devices may include, but are not limited to, smartphones, desktop computers, laptops, tablets, smart speakers, in-vehicle devices, and smartwatches. Servers may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing cloud computing services; this application does not impose specific limitations. Furthermore, the mentioned terminal devices and servers can be directly or indirectly connected via wired or wireless communication; this application does not impose specific limitations on these connections.

[0038] The aforementioned summary generation device is capable of performing natural language processing. Natural language processing (NLP) is an important field within computer science and artificial intelligence. It studies various theories and methods that enable effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language, i.e., the language people use in daily life, and thus it has a close relationship with linguistic research. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robot question answering, knowledge graphs, and other technologies. In the embodiments of this application, the summary generation device can process the text information in the document to be processed using text preprocessing, semantic understanding, and other technologies in natural language processing.

[0039] Furthermore, this abstract generation device can also possess computer vision processing capabilities. Computer vision, as mentioned, is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes for target recognition, trajectory tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition. In this embodiment, the abstract generation device can process images in the document to be processed using image processing and image semantic understanding techniques within computer vision technology.

[0040] Furthermore, this summarization generator also possesses machine learning capabilities. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as neural networks.

[0041] The summary generation method provided in this application uses an artificial intelligence model, mainly involving the application of neural networks, which predict and process the summary information of the document to be processed.

[0042] Because related solutions typically employ a deep Transformer network architecture to process the text information and all related images in a document in order to generate a summary of the document. Figure 1 A schematic diagram of the architecture for generating summary information in existing schemes is shown. For example... Figure 1As shown, text features are obtained by extracting features from the text information in the document, and visual features are obtained by extracting visual features from all the images in the document. Then, without changing the existing Transformer architecture, the text features and all visual features are simply concatenated and fused, and the concatenated and fused feature vector is decoded by a decoder to generate the corresponding summary information of the document. However, because all images in the document need to be considered, without considering whether each image contributes to the generated summary, the subsequent feature fusion process includes many useless visual features corresponding to images. As a result, the generated summary information is insufficient to accurately express the main content of the document, reducing the accuracy of the summary.

[0043] Based on this, in order to solve the aforementioned technical problems, this application provides a method for generating summaries. This method can be applied to application scenarios such as news headline generation, scientific literature summary generation, search result snippet generation, and product review summary generation; however, this application does not impose specific limitations. To facilitate understanding of the technical solution of this application, the following description, in conjunction with practical application scenarios, uses only a terminal device as the summary generation device to introduce the summary generation method provided in this application embodiment. In practical applications, devices such as servers can also be used as summary generation devices to execute the summary generation method provided in this application embodiment; however, this application does not impose specific limitations.

[0044] Figure 2 The illustration shows an application scenario diagram of the abstract generation method provided in the embodiments of this application. For example... Figure 2 The application scenarios shown include terminal devices. For example, the target object can obtain the document to be processed through a terminal device. Figure 2 In the scenario shown, the document to be processed includes text information and at least one image. From Figure 2 The description suggests that the text message roughly reads: "In the royal palace in the city center of Country A, mourners from rural areas wait to enter and sign condolence letters. Palace staff have arranged free public transportation for the mourners, and the Crown Prince and Princess have also paid their respects at the palace. On Friday, the King's body was transported by a procession from the hospital where he died to the Buddhist temple within the palace, and cremation is not expected for several months. The streets are crowded with mourners, many weeping as the procession passes. The King has earned the recognition of the people of Country A for his charitable acts in helping the poor in rural areas, and because Country A remained under control after a coup d'état in a certain year, he is seen as a stable leader by his subjects in a country frequently plagued by instability." Furthermore, corresponding illustrations are shown to correspond with the content of this text message, such as... Figure 2 The five images shown.

[0045] The terminal device performs feature extraction on the text information and five images in the document to be processed, thereby extracting corresponding text feature vectors and image feature vectors. Then, the terminal device performs self-attention processing on the extracted text feature vectors, as well as on the first text feature vector and image feature vectors. This allows the subsequent model to focus on key text and key images, discarding irrelevant content from images, and using the question feature vectors of key text and the image feature vectors of key images to determine the target feature vector. In this way, the terminal device, based on the summary generation model, predicts the target feature vector and the first summary words, accurately generating the target summary information for the document to be processed. For example, the target summary information could be: In the royal palace in the center of country A, mourners lined up to pay their respects to the king who died on Thursday.

[0046] It should be noted that the above Figure 2 The application scenarios shown are merely illustrative; in practical applications, other application scenarios may also be included, and this application does not impose any limitations. Similarly, the document to be processed mentioned is merely an example; in practical applications, other documents to be processed may also be included, and this application does not impose any specific limitations. Furthermore, the described text information, number of images, etc., are merely illustrative examples and are not specifically limited in the embodiments of this application.

[0047] Figure 3 A schematic diagram of the system flow framework for abstract generation provided in an embodiment of this application is shown.

[0048] like Figure 3As shown, during the encoding process, after acquiring the text information and images from the document to be processed, the text information can be processed using a feature extraction model to obtain the corresponding text feature vector, and the images can be processed to obtain the corresponding image feature vector. Then, a self-attention mechanism is used to process the text feature vector to obtain the corresponding first text feature vector. Similarly, an image-aware self-attention mechanism is used to process the first text feature vector and the image feature vector to obtain the target feature vector. Furthermore, during the decoding process, a marked multi-head attention mechanism can be used to process the first summary word to obtain the historical summary feature vector. Then, a self-attention mechanism is used to process the target feature vector and the historical summary feature vector to obtain the first feature vector. This first feature vector is then used as input to the summary generation model to predict the probability of each word, thereby generating the target summary information of the document to be processed.

[0049] It should be noted that the self-attention mechanism and image-aware multi-head attention mechanism described above can include both multi-head and single-head self-attention mechanisms, and are not specifically limited in this embodiment. In subsequent embodiments, the self-attention mechanism and image-aware self-attention mechanism will only be described using the multi-head self-attention mechanism as an example. Furthermore, the specific processing procedure of the described image-aware self-attention mechanism can be found in the following sections. Figure 6 We will understand the content described in the text, but will not elaborate further here.

[0050] The method for generating abstracts provided in the embodiments of this application will now be described with reference to the accompanying drawings. Figure 4 A flowchart illustrating a method for generating abstracts according to an embodiment of this application is shown. Figure 4 As shown, the method for generating this summary may include the following steps:

[0051] 401. Perform feature extraction on the text information in the document to be processed to obtain the text feature vector corresponding to the text information.

[0052] In this example, the target object can obtain the document to be processed through a terminal device, such as the aforementioned... Figure 2The document to be processed shown can be a scientific or technological document, or a product review document, etc., and is not specifically limited in this embodiment. After the target object obtains the document to be processed through the terminal device, the terminal device performs feature extraction processing on the text information in the document to obtain the text feature vector corresponding to the text information. For example, after obtaining the document to be processed, the terminal device can determine the corresponding text information from the document. Then, at the embedding layer, the terminal device can obtain the semantic vector (token embeddings) and position vector (position embeddings) of each word in the text information based on the feature extraction model, and fuse the semantic vector and the corresponding position vector of each word to obtain the text feature vector corresponding to each word. In this way, based on the text feature vectors corresponding to all words, the text feature vector corresponding to the text information can be obtained. It should be noted that the feature extraction model described may include, but is not limited to, long short-term memory (LSTM), recurrent neural network (RNN), convolutional neural network (CNN), transformer, etc., but no specific limitation is made in the embodiments of this application.

[0053] 402. Perform feature extraction processing on the images in the document to be processed to obtain the image feature vectors corresponding to the images.

[0054] In this example, after acquiring the document to be processed, the terminal device can also determine the illustrations in the document, i.e., obtain the corresponding images. Then, the terminal device performs feature extraction processing on the images in the document to be processed, thereby extracting the image feature vectors corresponding to the images. The described image feature vectors can also be called image visual feature vectors, etc., and this application does not make a specific limitation. It should be noted that the number of images described in this application can be one or more, and this application embodiment does not make a specific limitation.

[0055] For example, at the embedding layer, the terminal device can use feature extraction models such as Faster R-CNN to extract features from the images in the document to be processed, obtaining object embeddings, object ID embeddings, ROI box embeddings, and image ID embeddings for each object in the image. Then, the terminal device concatenates the object embeddings, object ID embeddings, ROI box embeddings, and image ID embeddings to obtain the corresponding image feature vector. For example... Figure 5 This diagram illustrates the extraction of image feature vectors according to an embodiment of this application. Figure 5 As shown, taking three images (such as images 1 to 3) as an example, for image 1, we can obtain the object feature vector, the identifier feature vector, the boundary feature vector, and the identifier feature vector of each object in image 1. These three vectors are then concatenated to obtain the image feature vector of image 1. Similarly, the process of obtaining the image feature vectors for images 2 and 3 can be understood by referring to the process for obtaining the image feature vector for image 1, which will not be elaborated here. After obtaining the image feature vectors for images 1 to 3, we can sum the image feature vectors of image 1, image 2, and image 3 to obtain the image feature vector corresponding to the image in the document to be processed. Obtaining the image feature vectors in this way preserves the order information between multiple images.

[0056] It should be noted that the execution order of steps 401 and 402 is not specifically limited in this embodiment. In practical applications, step 402 can be executed first and then step 401, or steps 401 and 402 can be executed simultaneously; this application does not impose any restrictions.

[0057] 403. Perform self-attention processing on the text feature vector to obtain the first text feature vector.

[0058] In this example, after extracting features from the text information in the document to be processed to obtain the corresponding text feature vector, the terminal device can perform self-attention processing on the text feature vector through a multi-head attention mechanism to obtain the first text feature vector. For example, the terminal device can correlate this text feature vector with the weight matrix W. q W k W v Weighted processing is performed to obtain the corresponding text query matrix, text key matrix, and text transpose matrix. Then, the terminal device applies self-attention processing to these matrices based on this self-attention mechanism to obtain the first text feature vector. The self-attention mechanism described can be understood with reference to the following formula:

[0059] X1 text =Multihead(W q X text W k X text W v X text )

[0060] Among them, W q W k W v These are the weight matrices, X and X. text W is the text feature vector. q X text W is a text query matrix. k X text W is a text key matrix. v X text X1 is the transpose of the text matrix. text This is the first text feature vector.

[0061] It should be noted that if the sequence length of the text information is s and the dimension of each word is d, then the sequence of word vectors input into the self-attention mechanism constitutes an R... s×d Matrix, i.e., text feature vector X text ∈R (s×d) At this point, this matrix is ​​compared with the three matrices W respectively. q W k W v ∈R d×d Perform matrix multiplication (i.e., linear transformation) to obtain three matrices Q, K, V∈R. s×d As the initial data for the dot product self-attention operation in the multi-head self-attention mechanism, the first text feature vector X1 is then calculated. text ∈Rs×d Among them, W q W k W v These are the weight matrices of the three trained neural networks, which are obtained by training the neural networks.

[0062] 404. Perform self-attention processing on the first text feature vector and the image feature vector to obtain the target feature vector.

[0063] In this example, after the terminal device extracts the first text feature vector in step 403, it can use the first text feature vector and the image feature vector as inputs to the image perception self-attention mechanism. The image perception self-attention mechanism then performs self-attention processing on the first text feature vector and the image feature vector to obtain the target feature vector C, i.e., C = FFN(Multihead([X1 text ;X img ],X1 text X1 text ), where X1 text Let X represent the first text feature vector. img This can represent image feature vectors; see below for details. Figure 6 The self-attention processing procedure shown is for understanding purposes only and will not be elaborated upon here. The target feature vector expresses more effective image and text information in the document to be processed. For example, the terminal device can fuse the first text feature vector with the image feature vector to obtain a fused feature vector. Then, the terminal device performs self-attention processing on the fused feature vector and the first text feature vector using an image-aware self-attention mechanism to obtain the target feature vector.

[0064] In some examples, the terminal device performs self-attention processing on the fused feature vector and the first text feature vector using an image-aware self-attention mechanism to obtain the target feature vector. This can be understood as follows: The terminal device performs self-attention processing on the first text feature vector output by the (i-1)th encoding layer in the encoder to obtain the first text feature vector of the i-th encoding layer, where 1 < i ≤ L, and L is the total number of encoding layers in the encoder. Then, the terminal device performs self-attention processing on the first text feature vector of the i-th encoding layer and the fused feature vector output by the (i-1)th layer in the encoder to obtain the fused feature vector of the i-th encoding layer. In this way, the terminal device can obtain the target feature vector by performing self-attention processing on the fused feature vector of the i-th encoding layer and the first text feature vector of the i-th encoding layer in the encoder. By using the above method, through the interactive iteration of the L-layer encoding layer in the encoder, the multi-head attention processing operation between the fused feature vector and the first text feature vector is completed. This fully extracts the key information in the image, providing richer and more effective image information for subsequent models, thereby discarding useless image information.

[0065] In other examples, the terminal device can determine the query matrix in the image-aware self-attention mechanism based on the fused feature vector output by the i-th encoding layer and a preset first weight matrix; and the terminal device can determine the key matrix in the image-aware self-attention mechanism based on the first text feature vector output by the i-th encoding layer and a second weight matrix, and the terminal device can determine the transpose matrix in the image-aware self-attention mechanism based on the first text feature vector output by the i-th encoding layer and a preset third weight matrix. Then, the terminal device calculates the target feature vector based on the query matrix, the key matrix, and the transpose matrix. For example, the terminal device can determine the target feature vector based on the query matrix, the key matrix, and the transpose matrix by calculating the similarity between the fused feature vector output by the i-th encoding layer and the first text feature vector output by each i-th encoding layer based on the query matrix and the key matrix; then, a weighted summation is performed based on each similarity and the transpose matrix to determine the target feature vector.

[0066] For example, Figure 6 A schematic diagram illustrating the processing flow of the image-aware self-attention mechanism provided in an embodiment of this application is shown. Figure 6 As shown, if the number of images obtained is p (p≥1, and p is an integer), and the dimension of each image is d, then the extracted image feature vector can be represented as X. img ∈R p×d After obtaining the first text feature vector and image feature vector Ximg ∈R p×d Then, the first text feature vector X1 can be... text ∈R s×d With image feature vector X img ∈R p×d The fusion process is performed to obtain the fused feature vector. Then, the fused feature vectors With the preset first weight matrix W Q Perform matrix multiplication to obtain the corresponding query matrix Q∈R (s+p)×d Similarly, the first text feature vector X1 text ∈R s×d and the preset second weight matrix W K Perform matrix multiplication to calculate the corresponding key matrix K∈R s×d and the first text feature vector X1 text ∈R s×d and the preset third weight matrix W V Perform matrix multiplication to calculate the corresponding transpose matrix V∈R s×d Therefore, further based on the query matrix Q∈R (s+p)×d Bond matrix K∈R s×d Calculate the similarity between the fused feature vector and each first text feature vector; for example, the query matrix Q∈R can be used. (s+p)×d With the key matrix K∈R s×d The similarity is calculated by multiplying the transpose of the matrix, i.e.: in, Represent the fused feature vector The nth fusion feature vector With the first text feature vector X1 text ∈R s×d The m-th eigenvector Similarity between them Represent the query matrix Q, Let K represent the key matrix, where d is an adjustable parameter. Thus, by assigning each similarity... With the transpose matrix V∈R s×d By performing a weighted summation, we can obtain the target feature vector, i.e. Among them, C n This represents the nth target feature vector. The transpose matrix V is expressed. Then, by further processing each of the n fused feature vectors with each of the first text feature vectors using the aforementioned image-aware self-attention mechanism, the final target feature vector C can be obtained, i.e.:

[0067] C = FFN(Multihead([X1 text ;X img ],X1 text X1 text )).

[0068] It should be noted that the described nth target feature vector C n This refers to a feature vector that is part of the final target feature vector C. The sequence length of the target feature vector C is s+p, and the dimension is d, which can be represented as C∈R. (s+p)×d Additionally, the matrix W mentioned... Q W K W V These are the weight matrices of the three trained neural networks, which are obtained by training the neural networks.

[0069] 405. Based on the summary generation model, predict the target feature vector and the first summary word to generate the target summary information of the document to be processed. The first summary word is the historical summary word that has been predicted in the target summary information of the document to be processed.

[0070] In this example, after the terminal device obtains the target feature vector through step 404, it can combine it with the first summary word as input to the summary generation model. The model then predicts the target feature vector and the first summary word to generate the target summary information for the document to be processed. It should be noted that the first summary word can be understood as a historical summary word that has already been predicted in the target summary information.

[0071] For example, in generating the t-th word Y in the target summary information t At this time, the terminal device can first obtain the historical summary words that have been predicted in the target summary information, that is, the first summary word Y. t-1 Then, the terminal device obtains the word embedding feature representation Y corresponding to the first summary word based on the first summary word, and uses a self-attention mechanism to embed the first summary word Y. t-1 The corresponding word embedding feature representation Y undergoes self-attention processing to obtain the historical summary feature vector, i.e., H. Y =Multihead(Y,Y,Y), where Y is the word embedding feature representation, H Y This is the feature vector for the historical summary.

[0072] In this way, the terminal device obtains the historical summary feature vector H Y Then, another self-attention mechanism is used to analyze the target feature vector and the historical summary feature vector H. Y Perform self-attention processing to obtain the first feature vector Z. L ZL =FFN(Multihead(H Y H(C,C), where C is the target feature vector and H(C,C) is the target feature vector. Y This is the feature vector for the historical summary. It should be noted that this first feature vector Z... L This can be understood as the target feature vector C and the historical summary feature vector H. Y The features and interactions between them are represented.

[0073] The terminal device uses this first feature vector as input to the summary generation model to obtain the predicted probability p(Y) of each word. t |x,O,Y<t), that is, p(Y t |x,O,Y<t)=soft max(W o Z L +b o ), where W o b o All parameters are adjustable. Finally, the terminal device generates the target summary information of the document to be processed based on the predicted probability of each word.

[0074] In other examples, the predicted probabilities of all words can be summed to obtain the target loss value L. MAS ,Right now Furthermore, the initial summary generation model is updated with parameters based on the target loss value to obtain the summary generation model.

[0075] In this embodiment, feature extraction is performed on the text information in the document to be processed to obtain the text feature vector corresponding to the text information, and feature extraction is performed on the images in the document to obtain the image feature vector corresponding to the images. Then, self-attention processing is further applied to the text feature vector to obtain the first text feature vector, enabling the subsequent model to focus on the key content in the text information. Self-attention processing is also performed on the first text feature vector and the image feature vector to obtain the target feature vector, further enabling the subsequent model to focus on the key text and key images in the text information and images, thereby discarding the content in useless images. The target feature vector can be determined using the question feature vector of the key text and the image feature vector of the key images, ensuring that the obtained target feature vector does not include the image feature vector corresponding to useless images. Thus, based on the summarization generation model, the target feature vector and the first summary words are predicted, accurately generating the target summary information of the document to be processed, ensuring that the target summary information accurately expresses the main content required by the document, and improving the accuracy of the summary.

[0076] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. It is understood that to achieve the above functions, corresponding hardware structures and / or software modules are included to execute each function. Those skilled in the art should readily recognize that, based on the modules and algorithm steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0077] This application embodiment can divide the device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0078] The abstract generation apparatus in the embodiments of this application will now be described in detail. Figure 7 This is a schematic diagram of one embodiment of the abstract generation apparatus provided in this application. Figure 7 As shown, the summary generation apparatus may include a processing unit 701 and an acquisition unit 702.

[0079] The processing unit 701 is used to perform feature extraction processing on the text information in the document to be processed, to obtain the text feature vector corresponding to the text information, and to perform feature extraction processing on the images in the document to be processed, to obtain the image feature vector corresponding to the images. The images are used to indicate the illustrations in the document to be processed. For details, please refer to the foregoing. Figure 4 The content of steps 401 to 402 in the previous section will be understood and will not be repeated here.

[0080] Processing unit 701 performs self-attention processing on the text feature vector to obtain a first text feature vector. For details, please refer to the foregoing. Figure 4 The content of step 403 in the previous section will be understood and will not be repeated here.

[0081] Processing unit 701 performs self-attention processing on the first text feature vector and the image feature vector to obtain the target feature vector. For details, please refer to the foregoing. Figure 4 The content of step 404 in the previous section will be understood and will not be repeated here.

[0082] The processing unit 701 is used to predict the target feature vector and the first summary word based on a preset summary generation model, and generate the target summary information of the document to be processed. The first summary word is the historical summary word that has been predicted in the target summary information of the document to be processed. For details, please refer to the foregoing. Figure 4 The content of step 405 in the previous section will be understood and will not be repeated here.

[0083] In some optional examples, the processing unit 701 is used to: fuse the first text feature vector with the image feature vector to obtain a fused feature vector; and perform self-attention processing on the fused feature vector and the first text feature vector to obtain a target feature vector.

[0084] In some alternative examples, the processing unit 701 is configured to: perform self-attention processing on the first text feature vector output by the (i-1)th encoding layer in the encoder to obtain the first text feature vector of the i-th encoding layer, where 1 < i ≤ L, L is a natural number, and L is the total number of encoding layers in the encoder; perform self-attention processing on the first text feature vector of the i-th encoding layer and the fused feature vector output by the (i-1)th encoding layer in the encoder to obtain the fused feature vector of the i-th encoding layer; and perform self-attention processing on the fused feature vector of the i-th encoding layer and the first text feature vector of the i-th encoding layer in the encoder to obtain the target feature vector.

[0085] In some alternative examples, the processing unit 701 is used to: determine the query matrix in the image-aware self-attention mechanism based on the fused feature vector output by the i-th encoding layer and a preset first weight matrix; determine the key matrix in the image-aware self-attention mechanism based on the first text feature vector output by the i-th encoding layer and a preset second weight matrix; and determine the transpose matrix in the image-aware self-attention mechanism based on the first text feature vector output by the i-th encoding layer and a preset third weight matrix; and determine the target feature vector based on the query matrix, the key matrix, and the transpose matrix.

[0086] In some alternative examples, the processing unit 701 is used to: calculate the similarity between the fused feature vector output by the i-th coding layer and the first text feature vector output by each i-th coding layer, based on the query matrix and the key matrix; and perform a weighted summation based on each similarity and the transpose matrix to determine the target feature vector.

[0087] In some alternative examples, the processing unit 701 is used to: perform self-attention processing on the first summary words to obtain historical summary feature vectors; perform self-attention processing on the target feature vector and the historical summary feature vectors to obtain a first feature vector; use the first feature vector as input to the summary generation model to obtain the predicted probability of each word; and generate target summary information of the document to be processed based on the predicted probability of each word.

[0088] In some alternative examples, the processing unit 701 is also used to: sum the predicted probabilities of each word to obtain a target loss value; and update the parameters of the initial summary generation model based on the target loss value to obtain a summary generation model.

[0089] In some alternative examples, the acquisition unit 702 is used to: acquire the word embedding feature representation corresponding to the first summary word. The processing unit 701 is used to: perform self-attention processing on the word embedding feature representation corresponding to the first summary word to obtain the historical summary feature vector.

[0090] In some alternative examples, the acquisition unit 702 is used to: acquire the semantic vector and position vector of each word in the text information of the document to be processed. The processing unit 701 is used to: fuse the semantic vector and position vector of each word to obtain the text feature vector corresponding to the text information.

[0091] In some alternative examples, the acquisition unit 702 is used to: perform feature extraction processing on the images in the document to be processed, and acquire the object feature vector, the identifier feature vector, the boundary feature vector of each object, and the identifier feature vector of the image for each object in the image. The processing unit 701 is used to: concatenate the object feature vector, the identifier feature vector, the boundary feature vector, and the identifier feature vector of the image for each object to obtain the image feature vector corresponding to the image.

[0092] The summary generation device in the embodiments of this application has been described above from the perspective of modular functional entities. The summary generation device in the embodiments of this application is described below from the perspective of hardware processing. Figure 8 This is a schematic diagram of the structure of the abstract generation apparatus provided in the embodiments of this application. The abstract generation apparatus can vary considerably due to differences in configuration or performance. The abstract generation apparatus may include at least one processor 801, a communication line 807, a memory 803, and at least one communication interface 804.

[0093] The processor 801 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (server IC), or one or more integrated circuits used to control the execution of the program of the present application.

[0094] Communication line 807 may include a path for transmitting information between the aforementioned components.

[0095] The communication interface 804 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0096] The memory 803 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions. The memory can exist independently and be connected to the processor via communication line 807. The memory can also be integrated with the processor.

[0097] The memory 803 stores computer execution instructions for implementing the scheme of this application, and its execution is controlled by the processor 801. The processor 801 executes the computer execution instructions stored in the memory 803, thereby implementing the method provided in the above embodiments of this application.

[0098] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0099] In a specific implementation, as one example, the summary generation apparatus may include multiple processors, such as... Figure 8 Processors 801 and 802 are described herein. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0100] In a specific implementation, as one embodiment, the summary generation apparatus may further include an output device 805 and an input device 806. The output device 805 communicates with the processor 801 and can display information in various ways. The input device 806 communicates with the processor 801 and can receive input from the target object in various ways. For example, the input device 806 may be a mouse, a touch screen device, or a sensing device, etc.

[0101] The aforementioned digest generation device can be a general-purpose device or a dedicated device. In specific implementations, the digest generation device can be a server, terminal equipment, or other similar devices. Figure 8 A device with a similar structure. The embodiments of this application do not limit the type of this abstract generation device.

[0102] It should be noted that Figure 8 The processor 801 in the memory can call computer execution instructions stored in the memory 803 to cause the digest generation device to perform actions such as... Figure 4 The method in the corresponding method embodiment.

[0103] Specifically, Figure 7 The function / implementation process of the processing unit 701 can be achieved through... Figure 8 The processor 801 in the memory calls computer execution instructions stored in the memory 803 to implement the function. Figure 7 The function / implementation process of the acquisition unit 702 can be achieved through... Figure 8 It is implemented using the 804 communication interface.

[0104] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the abstract generation methods described in the above method embodiments.

[0105] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the digest generation methods described in the above method embodiments.

[0106] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0107] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0109] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, they can be implemented in whole or in part in the form of a computer program product.

[0113] A computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, they generate, in whole or in part, the processes or functions according to embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., SSDs), etc.

[0114] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating summaries, characterized in that, include: The text information in the document to be processed is subjected to feature extraction processing to obtain the text feature vector corresponding to the text information, and the images in the document to be processed are subjected to feature extraction processing to obtain the image feature vector corresponding to the images. The text feature vector is subjected to self-attention processing to obtain the first text feature vector; The first text feature vector and the image feature vector are fused to obtain a fused feature vector; By performing self-attention processing on the first text feature vector output by the (i-1)th encoding layer in the encoder, the first text feature vector of the i-th encoding layer is obtained, where 1 < i ≤ L, L is a natural number, and L is the total number of encoding layers in the encoder. By performing self-attention processing on the first text feature vector of the i-th coding layer and the fused feature vector output by the (i-1)-th coding layer in the encoder, the fused feature vector of the i-th coding layer is obtained. By performing self-attention processing on the fused feature vector of the i-th encoding layer and the first text feature vector of the i-th encoding layer through the encoder, the target feature vector is obtained. Based on the summary generation model, the target feature vector and the first summary word are predicted to generate the target summary information of the document to be processed. The first summary word is a historical summary word that has been predicted in the target summary information of the document to be processed.

2. The method according to claim 1, characterized in that, The step of performing self-attention processing on the fused feature vector of the i-th encoding layer and the first text feature vector of the i-th encoding layer to obtain the target feature vector includes: Based on the fused feature vector output by the i-th coding layer and the preset first weight matrix, the query matrix in the image perception self-attention mechanism is determined; Based on the first text feature vector output by the i-th encoding layer and the preset second weight matrix, the key matrix in the image perception self-attention mechanism is determined, and based on the first text feature vector output by the i-th encoding layer and the preset third weight matrix, the transpose matrix in the image perception self-attention mechanism is determined. The target feature vector is determined based on the query matrix, the key matrix, and the transpose matrix.

3. The method according to claim 2, characterized in that, Determining the target feature vector based on the query matrix, the key matrix, and the transpose matrix includes: Based on the query matrix and the key matrix, calculate the similarity between the fused feature vector output by the i-th coding layer and the first text feature vector output by each i-th coding layer; The target feature vector is determined by performing a weighted summation process based on each similarity score and the transpose matrix.

4. The method according to any one of claims 1 to 3, characterized in that, The step of predicting the target feature vector and the first summary words based on the summary generation model to generate target summary information for the document to be processed includes: Self-attention processing is applied to the first summary words to obtain the historical summary feature vector; Self-attention processing is performed on the target feature vector and the historical summary feature vector to obtain a first feature vector; The first feature vector is used as the input to the summary generation model to obtain the predicted probability of each word; The target summary information of the document to be processed is generated based on the predicted probability of each word.

5. The method according to claim 4, characterized in that, The method further includes: The predicted probabilities of each word are summed to obtain the target loss value; The initial summary generation model is updated based on the target loss value to obtain the summary generation model.

6. The method according to claim 4, characterized in that, The self-attention processing of the first summary words to obtain the historical summary feature vector includes: Based on the first summary word, obtain the word embedding feature representation corresponding to the first summary word; Self-attention processing is performed on the word embedding feature representation corresponding to the first summary word to obtain the historical summary feature vector.

7. The method according to any one of claims 1 to 3, characterized in that, The step of performing feature extraction processing on the text information in the document to be processed to obtain the text feature vector corresponding to the text information includes: Based on the text information in the document to be processed, obtain the semantic vector and position vector of each word in the text information; The semantic vector and position vector of each word are fused to obtain the text feature vector corresponding to the text information.

8. The method according to any one of claims 1 to 3, characterized in that, The step of performing feature extraction processing on the images in the document to be processed to obtain the image feature vectors corresponding to the images includes: Feature extraction processing is performed on the images in the document to be processed to obtain the object feature vector of each object in the image, the identification feature vector of each object, the boundary feature vector of each object, and the identification feature vector of the image. The object feature vector, the identifier feature vector, the boundary feature vector of each object, and the identifier feature vector of the image are concatenated to obtain the image feature vector corresponding to the image.

9. A summary generation apparatus, characterized in that, include: The processing unit is used to perform feature extraction processing on the text information in the document to be processed to obtain the text feature vector corresponding to the text information, and to perform feature extraction processing on the image in the document to be processed to obtain the image feature vector corresponding to the image, wherein the image is used to indicate the illustration situation in the document to be processed; The processing unit is used to perform self-attention processing on the text feature vector to obtain a first text feature vector. The processing unit is configured to fuse the first text feature vector with the image feature vector to obtain a fused feature vector; perform self-attention processing on the first text feature vector output by the (i-1)th encoding layer in the encoder to obtain the first text feature vector of the i-th encoding layer, where 1 < i ≤ L, L is a natural number, and L is the total number of encoding layers in the encoder; perform self-attention processing on the first text feature vector of the i-th encoding layer and the fused feature vector output by the (i-1)th encoding layer in the encoder to obtain the fused feature vector of the i-th encoding layer; and perform self-attention processing on the fused feature vector of the i-th encoding layer and the first text feature vector of the i-th encoding layer in the encoder to obtain the target feature vector. The processing unit is used to perform prediction processing on the target feature vector and the first summary word based on a preset summary generation model to generate target summary information of the document to be processed, wherein the first summary word is a historical summary word that has been predicted in the target summary information of the document to be processed.

10. The apparatus according to claim 9, characterized in that, The processing unit is specifically used for: Based on the fused feature vector output by the i-th coding layer and the preset first weight matrix, the query matrix in the image perception self-attention mechanism is determined; Based on the first text feature vector output by the i-th encoding layer and the preset second weight matrix, the key matrix in the image perception self-attention mechanism is determined, and based on the first text feature vector output by the i-th encoding layer and the preset third weight matrix, the transpose matrix in the image perception self-attention mechanism is determined. The target feature vector is determined based on the query matrix, the key matrix, and the transpose matrix.

11. The apparatus according to claim 10, characterized in that, The processing unit is specifically used for: Based on the query matrix and the key matrix, calculate the similarity between the fused feature vector output by the i-th coding layer and the first text feature vector output by each i-th coding layer; The target feature vector is determined by performing a weighted summation process based on each similarity score and the transpose matrix.

12. The apparatus according to any one of claims 9 to 11, characterized in that, The processing unit is specifically used for: Self-attention processing is applied to the first summary words to obtain the historical summary feature vector; Self-attention processing is performed on the target feature vector and the historical summary feature vector to obtain a first feature vector; The first feature vector is used as the input to the summary generation model to obtain the predicted probability of each word; The target summary information of the document to be processed is generated based on the predicted probability of each word.

13. The apparatus according to claim 12, characterized in that, The processing unit is further configured to: The predicted probabilities of each word are summed to obtain the target loss value; The initial summary generation model is updated based on the target loss value to obtain the summary generation model.

14. The apparatus according to claim 12, characterized in that, The device further includes an acquisition unit; The acquisition unit is specifically used to acquire the word embedding feature representation corresponding to the first summary word based on the first summary word; The processing unit is specifically used to perform self-attention processing on the word embedding feature representation corresponding to the first summary word to obtain the historical summary feature vector.

15. The apparatus according to any one of claims 9 to 11, characterized in that, The device further includes an acquisition unit; The acquisition unit is specifically used to acquire the semantic vector and position vector of each word in the text information based on the text information in the document to be processed; The processing unit is specifically used to fuse the semantic vector and position vector of each word to obtain the text feature vector corresponding to the text information.

16. The apparatus according to any one of claims 9 to 11, characterized in that, The device further includes an acquisition unit; The acquisition unit is specifically used to perform feature extraction processing on the images in the document to be processed, and to acquire the object feature vector of each object in the image, the identification feature vector of each object, the boundary feature vector of each object, and the identification feature vector of the image. The processing unit is specifically used to concatenate the object feature vector of each object, the identifier feature vector of each object, the boundary feature vector of each object, and the identifier feature vector of the image to obtain the image feature vector corresponding to the image.

17. A summary generation apparatus, characterized in that, The summary generation device includes: an input / output (I / O) interface, a processor, and a memory, wherein the memory stores program instructions; The processor is configured to execute program instructions stored in the memory to perform the method as described in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer device or processor, cause the computer device or processor to perform the method as described in any one of claims 1 to 8.

Images