Text summary generation method, device, equipment and medium

By aligning the image and text features, a more accurate text summary is generated, which solves the problem of failing to effectively consider the alignment relationship between picture information and text information in the prior art, and improves the accuracy of the summary.

CN114741498BActive Publication Date: 2024-11-08PING AN TECH (SHENZHEN) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210422074.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-21
Publication Date
2024-11-08
Estimated Expiration
2042-04-21

AI Technical Summary

Technical Problem

The existing automatic digest technology fails to effectively consider the alignment relationship between image information and text information when generating text summary, resulting in inaccurate summary information.

Method used

By segmenting the images in the to-processed document into multiple images, using a preset transformer model to extract image features, and combining the word vector features after word segmentation for alignment, the alignment of image and text features is enhanced by using contrast learning and attention mechanisms. , and finally input the decoder to generate a summary.

Benefits of technology

Improves the accuracy of text summary, so that the summary more accurately reflects the associated information between pictures and text in the document.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114741498B_ABST
    Figure CN114741498B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of artificial intelligence technology, and provides a text summary generation method, device, equipment and medium. The method includes: dividing a first image of a document to be processed into multiple second images and inputting them into a preset transformer model for patch embedding, and outputting the image features of each second image; segmenting the text of the document to be processed according to a preset word segmentation algorithm to obtain multiple word segments and inputting them into a preset transformer model for tag embedding, and outputting the word vector features of each word segment; inputting the image features of each second image and the word vector features of each word segment into a preset alignment model for alignment processing, and outputting the alignment result; inputting the alignment result into a decoder of a preset transformer model, outputting multiple keywords corresponding to the document to be processed, splicing all the keywords in order, and inputting the summary of the document to be processed. The present invention also relates to the field of blockchain technology, and the above-mentioned image features and word vector features can also be stored in a node of a blockchain.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a text summary generation method, device, equipment and medium. Background Art

[0002] Automatic summarization is a technology that uses computer systems to automatically analyze text, summarize content, and automatically generate summaries. Automatic summarization technology can effectively help users find interesting content from retrieved articles and improve reading speed and quality.

[0003] In the existing traditional technology, text information is mostly used, and the text content is simply used to summarize the article. There are two types of summaries: extractive and generative. The core idea of ​​the extractive method is to calculate the importance of sentences in the text, such as the method of calculating the tf-idf value. The generative method generally uses the seq2seq model, inputs the entire article, outputs the summary sentence of this article, or inputs important words in the article and outputs the summary.

[0004] However, modern articles rarely contain pure text content. Instead, they are accompanied by pictures related to the article to present the content in a graphic and textual way. The pictures also contain relevant content information. In the process of generating summaries, the content information of the pictures also needs to be referred to. However, the alignment relationship between the picture information and the text information in actual situations is often not taken into account, and the resulting summary information is inaccurate. Summary of the invention

[0005] In view of the above, the present invention provides a text summary generation method, device, equipment and medium, which aims to solve the technical problem of inaccurate summary information of a document to be processed generated in the prior art.

[0006] To achieve the above object, the present invention provides a text summary generation method, the method comprising:

[0007] Splitting a first image contained in a document to be processed into a plurality of second images, inputting the plurality of second images into a preset transformer model for patch embedding, and outputting image features of each second image;

[0008] Segmenting the text of the document to be processed according to a preset segmentation algorithm to obtain multiple segmentations, inputting each segmentation into a preset transformer model for tag embedding, and outputting a word vector feature of each segmentation;

[0009] Inputting the image features of each of the second pictures and the word vector features of each of the word segments into a preset alignment model for alignment processing, and outputting the alignment result;

[0010] The alignment result is input into the decoder of the preset transformer model, a plurality of keywords corresponding to the document to be processed are output, all the keywords are concatenated in sequence, and a summary of the document to be processed is input.

[0011] Preferably, the step of inputting the image features of each of the second images and the word vector features of each of the word segments into a preset alignment model for alignment processing and outputting the alignment result comprises:

[0012] Input the image features of each of the second images and the word vector features of each of the segmented words into a preset alignment model to perform attention weight allocation, and output a first initial value of each of the second images and a second initial value of each of the segmented words;

[0013] Performing a pooling operation on the first initial value and the second initial value respectively to obtain a vector of each of the second pictures and a vector of each of the word segments;

[0014] According to the vector of each of the second images and the vector of each of the word segments, a preset number of cyclic alignments are performed on the image features of each of the second images and the word vector features of each of the word segments, and an alignment result is output.

[0015] Preferably, performing a preset number of cyclic alignments on the image features of each second picture and the word vector features of each word segmentation according to the vector of each second picture and the vector of each word segmentation, and outputting the alignment result, includes:

[0016] Performing a pooling operation on all the image features of the second image to obtain a vector of the first image, and performing a pooling operation on the word vector features of all the word segments to obtain a vector of the text;

[0017] According to a preset first loss value calculation formula, the vector of the first image and the vector of each of the second images are compared and trained to obtain a first loss value, and according to a preset second loss value calculation formula, the vector of the text and the vector of each of the word segments are compared and trained to obtain a second loss value;

[0018] According to the first loss value and the second loss value, the parameters of the preset alignment model in the back propagation are adjusted, and the image features of each of the second pictures and the word vector features of each of the word segments are aligned for a preset number of times, and the alignment result is output.

[0019] Preferably, the step of inputting the image features of each of the second images and the word vector features of each of the word segments into a preset alignment model for attention weight allocation, and outputting the first initial value of each of the second images and the second initial value of each of the word segments, comprises:

[0020] According to the attention mechanism of the preset alignment model, the image features of each of the second pictures and the word vector features of each of the word segments are assigned attention weights and similarity is calculated to obtain a first information vector corresponding to the image features of each of the second pictures, and output a second information vector corresponding to the word vector features of each of the word segments;

[0021] Input a first activation function and a second activation function, concatenate the first information vector and the second information vector, perform a preset number of alignment operations, and output a first initial value of each of the second images and a second initial value of each of the segmented words.

[0022] Preferably, the preset first loss value calculation formula includes:

[0023]

[0024] in, is the first loss value, K is the number of cycles of the preset alignment model, L is the number of samples sampled during the training of the preset alignment model, is the inner product value of the positive sample of the preset alignment model, is the sum of the inner product values ​​of the positive samples of the preset alignment model and all negative samples, Ι is the vector of the first image, T is the vector of the text, i is the vector of the i-th row of the first image matrix, k is the vector of the k-th row of the text matrix, and τ is a hyperparameter.

[0025] Preferably, the preset second loss value calculation formula includes:

[0026]

[0027] in, is the second loss value, K is the number of cycles of the preset alignment model, L is the number of samples sampled during the training of the preset alignment model, is the inner product value of the positive sample of the preset alignment model, is the sum of the inner product values ​​of the positive samples of the preset alignment model and all negative samples, Ι is the vector of the first image, T is the vector of the text, i is the vector of the i-th row of the first image matrix, k is the vector of the k-th row of the text matrix, and τ is a hyperparameter.

[0028] Preferably, the step of inputting the alignment result into a decoder of the preset transformer model and outputting a plurality of keywords corresponding to the document to be processed comprises:

[0029] A1. According to the arrangement order of each sentence of the text, at a first moment, input the alignment result into the decoder of the preset transformation model to generate a summary, thereby generating a first keyword;

[0030] A2. At a second moment, inputting the word vector of the first keyword and the alignment result into the decoder of the preset transformation model to generate a summary, thereby generating a second keyword;

[0031] A3. Repeat A1-A2 until the keywords of each sentence in the text are obtained.

[0032] To achieve the above object, the present invention further provides a text summary generating device, the device comprising:

[0033] An acquisition module: used for dividing a first image contained in a document to be processed into a plurality of second images, inputting the plurality of second images into a preset transformer model for patch embedding, and outputting image features of each second image;

[0034] Word segmentation module: used for segmenting the text of the document to be processed according to a preset word segmentation algorithm to obtain multiple word segments, inputting each word segment into a preset transformer model for tag embedding, and outputting word vector features of each word segment;

[0035] Output module: used for inputting the image features of each second picture and the word vector features of each word segmentation into a preset alignment model for alignment processing, and outputting the alignment result;

[0036] Generation module: used for inputting the alignment result into the decoder of the preset transformer model, outputting multiple keywords corresponding to the document to be processed, splicing all the keywords in order, and inputting the summary of the document to be processed.

[0037] To achieve the above object, the present invention further provides an electronic device, the electronic device comprising:

[0038] at least one processor; and,

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores a program executable by the at least one processor, and the program is executed by the at least one processor so that the at least one processor can execute the text summary generating method according to any one of claims 1 to 7.

[0041] To achieve the above-mentioned purpose, the present invention also provides a computer-readable medium, which stores text summary generation. When the text summary generation is executed by a processor, the steps of the text summary generation method as described in any one of claims 1 to 7 are implemented.

[0042] The present invention obtains the image features and text features of the document to be processed through a preset transformer model, inputs the image features and text features into a preset alignment model for alignment processing, and continuously aligns and enhances the important information in the image features and text features in combination with the attention mechanism during the cyclic alignment process, so that the summary result is more biased towards the association information between the image and text of the document to be processed. At the same time, a training method for contrastive learning is added, which can enable the preset alignment model to treat the combination differences of image features and text features more differently, so that the feature expressions of image information and text information converge and output alignment results, and the alignment results are input into the decoder of the preset transformer model for summary generation to obtain the summary of the document to be processed, thereby improving the accuracy of generating summary information of the document to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart diagram of a preferred embodiment of the text summary generation method of the present invention;

[0044] Figure 2 A schematic diagram of modules of a preferred embodiment of the text summary generating device of the present invention;

[0045] Figure 3 A schematic diagram of a preferred embodiment of the electronic device of the present invention;

[0046] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0048] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0049] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0050] The present invention provides a method for generating a text summary. Figure 1 FIG. 1 is a flow chart of a method for generating a text summary according to an embodiment of the present invention. The method can be executed by an electronic device, and the electronic device can be implemented by software and / or hardware. The text summary generation method includes:

[0051] Step S10: Segment the first image included in the document to be processed into multiple second images, input the multiple second images into a preset transformer model for patch embedding, and output image features of each second image.

[0052] In this embodiment, a document to be processed input by a user is obtained. The document to be processed includes a text image in which text and images are mixed. The text (words) of the document to be processed can accurately express semantic information. The image (first image) of the document to be processed can help users quickly obtain the subject. The information of the two modalities can complement each other. In order to obtain the aligned vectors of the text and the first image, the text and the first image of the document to be processed need to be obtained.

[0053] Based on an image segmentation algorithm (e.g., Image Segmentation), the first image is segmented into an image matrix containing multiple second images, for example, the first image is segmented into a 3X3 rectangle, containing a total of 9 second images. Each second image is input into a preset transformer model for patch embedding, the preset transformer model includes but is not limited to a Transformer model, and the Transformer model includes two architectures: an encoder and a decoder.

[0054] The encoder of the preset transformer model performs patch embedding on each second image input, specifically including: the preset transformer model receives the first image embedding of the 1D sequence as input, and transforms the first image x∈R^H×W×C into a series of flattened 2D images x_p∈R^N×(P^2·C), where (H, W) represents the resolution of the first image, and (P, P) represents the resolution of each second image. Then, N=HW / P^2 becomes the effective sequence length of the preset transformer model. The preset transformer model uses the same width in all layers, and a trainable linear projection maps the vectorized patch of each second image to the model dimension D, and the corresponding output is the image feature of each second image.

[0055] Step S20: Segment the text of the document to be processed according to a preset segmentation algorithm to obtain multiple segmentations, input each segmentation into a preset transformer model for tag embedding, and output the word vector feature of each segmentation.

[0056] In this embodiment, the preset word segmentation algorithm includes but is not limited to the Jieba word segmentation algorithm. Jieba word segmentation is a Python Chinese word segmentation component that can perform word segmentation, part-of-speech tagging, keyword extraction and other functions on Chinese text, and supports custom dictionaries. Each word segmentation is input into the preset transformer model for token embedding, specifically including: the preset transformer model receives the text sequence of each word segmentation as input, and the text sequence is represented by a matrix through the encoder to obtain the word vector feature of each word segmentation. For example, the text sequence of a word segmentation is read through the encoder as [3, 5, 7, 8, 4, 5, 4], the size of the dictionary is N, and the dimension of the vector is d, then the word vector feature of the word segmentation is a vector of [7, d].

[0057] When embedding each second picture and each word segmentation input into the preset transformer model, the structure of the preset transformer model is exactly the same, but the training parameters in the model are different. During initialization, the parameters of the two models are random respectively. In the process of using error gradient backpropagation, the parameters of the two models will not be shared or copied.

[0058] Step S30: input the image features of each of the second pictures and the word vector features of each of the word segments into a preset alignment model for alignment processing, and output the alignment result.

[0059] Specifically, step S30 includes:

[0060] Input the image features of each of the second pictures and the word vector features of each of the segmented words into a preset alignment model to perform attention weight allocation, and output a first initial value of each of the second pictures and a second initial value of each of the segmented words;

[0061] Performing a pooling operation on the first initial value and the second initial value respectively to obtain a vector of each of the second pictures and a vector of each of the word segments;

[0062] According to the vector of each of the second images and the vector of each of the word segments, a preset number of cyclic alignments are performed on the image features of each of the second images and the word vector features of each of the word segments, and an alignment result is output.

[0063] In one embodiment, performing a preset number of loop alignments on the image features of each second picture and the word vector features of each word segmentation according to the vector of each second picture and the vector of each word segmentation, and outputting the alignment result, includes:

[0064] Performing a pooling operation on all the image features of the second image to obtain a vector of the first image, and performing a pooling operation on the word vector features of all the word segments to obtain a vector of the text;

[0065] According to a preset first loss value calculation formula, the vector of the first image and the vector of each of the second images are compared and trained to obtain a first loss value, and according to a preset second loss value calculation formula, the vector of the text and the vector of each of the word segments are compared and trained to obtain a second loss value;

[0066] According to the first loss value and the second loss value, the parameters of the preset alignment model in the back propagation are adjusted, and the image features of each of the second pictures and the word vector features of each of the word segments are aligned for a preset number of times, and the alignment result is output.

[0067] In this embodiment, the preset alignment model includes but is not limited to the RA Layer cyclic alignment model. The present invention adopts the RALayer cyclic alignment model in combination with the attention mechanism to continuously align and enhance the important information in the image features and text features, so that the summary result is more biased towards the correlation information between the image and text of the document to be processed. The image features of each second image and the word vector features of each word segment are input into the preset alignment model for alignment processing. The number of cyclic alignments is K. For example, the number of cycles in the present invention is 6 times, and the first initial value of each second image and the second initial value of each word segment are output. Then, the first initial value and the second initial value are respectively subjected to pooling operations. The pooling operation includes adding and averaging the columns in the matrix of the first initial value and the second initial value, converting the matrix into a vector, thereby obtaining the vector of each second image and the vector of each word segment.

[0068] All the image features of the second image are also pooled to obtain the vector of the first image. That is to say, the maximum information value of all the image features of the second image is retained in the pooling template through the pooling operation, and the maximum information value of the word vector features of all the segmented words is retained in the pooling template.

[0069] Through the supervised learning module of comparative learning training, the similarity data between the vector of the entire first image and the vector of each second image is analyzed, the similarity data between the vector of the entire text and the vector of each word segment is analyzed, and the similarity data between the vector of the entire first image, each second image, the entire text and the vector of each word segment are cross-analyzed. Through the softmax classifier (the softmax classifier is a classifier with classified positive samples and negative samples), a group of data corresponding to the entire first image and the entire text is used as a positive sample, and the data that does not correspond to the entire first image and the entire text is used as a negative sample. The similarity between the positive and negative samples is calculated respectively, and the first loss value and the second loss value are obtained by using the vector inner product according to the similarity calculation. According to the first loss value and the second loss value, the parameters of the preset alignment model in the back propagation are adjusted, and the image features of each second image and the word vector features of each word segment are subjected to a preset number of cyclic alignments, and the alignment results are output.

[0070] In one embodiment, the inputting the image features of each second picture and the word vector features of each word segment into a preset alignment model for attention weight allocation, and outputting the first initial value of each second picture and the second initial value of each word segment, comprises:

[0071] According to the attention mechanism of the preset alignment model, the image features of each of the second pictures and the word vector features of each of the word segments are assigned attention weights and similarity is calculated to obtain a first information vector corresponding to the image features of each of the second pictures, and output a second information vector corresponding to the word vector features of each of the word segments;

[0072] Input a first activation function and a second activation function, splice the first information vector and the second information vector and perform a preset number of alignment operations, output a first initial value of each of the second pictures and a second initial value of each of the word segments, the first activation function is a sigmoid activation function, and the second activation function is a tanh activation function.

[0073] In one embodiment, the step of allocating attention weights and calculating similarity between the image features of each second picture and the word vector features of each word segment, obtaining a first information vector corresponding to the image features of each second picture, and outputting a second information vector corresponding to the word vector features of each word segment, includes:

[0074] Copy a preset number of image features of each of the second pictures and word vector features of each of the word segments to perform attention weight calculation to obtain a first similarity measure and a second similarity measure;

[0075] The first similarity measure and the second similarity measure are vector-valued regularized according to a third activation function to obtain a first information vector corresponding to the image feature of each of the second images, and a second information vector corresponding to the word vector feature of each of the word segments is output, wherein the third activation function is a ReLU activation function.

[0076] When allocating attention weights and calculating similarity for the image features of each second picture and the word vector features of each word segmentation, first copy a preset number of image features of the second picture and word vector features of the word segmentation, for example, first copy 3 copies, respectively as k, q, and v, and then use k and q to calculate the attention weights, and then multiply them with v to obtain the first similarity measure and the second similarity measure.

[0077] According to the third activation function (ReLU activation function), after the similarity calculation of the first similarity metric and the second similarity metric, the vector value is regularized, and then the attention mechanism is used to calculate and obtain which information of the second picture and the word segmentation has important information vectors and which information does not have important information vectors, thereby outputting the first information vector corresponding to the image features of each second picture and obtaining the second information vector corresponding to the word vector features of each word segmentation.

[0078] Input the first activation function (sigmoid activation function) and the second activation function (tanh activation function), concatenate the first information vector and the second information vector, and perform a preset number of alignment operations, so that you can know which important information needs to be remembered and which information needs to be forgotten. For example, the information X* that needs to be remembered will be used as a new input X and input into the next loop, and this loop will be executed K times. The image features of the second picture and the word vector features of the word segmentation are aligned K times by the preset alignment model. For example, after alignment 6 times, the first initial value of each of the second pictures and the second initial value of each of the word segments are output.

[0079] In one embodiment, the preset first loss value calculation formula includes:

[0080]

[0081] in, is the first loss value, K is the number of cycles of the preset alignment model, L is the number of samples sampled during the training of the preset alignment model, is the inner product value of the positive sample of the preset alignment model, is the sum of the inner product values ​​of the positive samples of the preset alignment model and all negative samples, Ι is the vector of the first image, T is the vector of the text, i is the vector of the i-th row of the first image matrix, k is the vector of the k-th row of the text matrix, and τ is a hyperparameter.

[0082] In one embodiment, the preset second loss value calculation formula includes:

[0083]

[0084] in, is the second loss value, K is the number of cycles of the preset alignment model, L is the number of samples sampled during the training of the preset alignment model, is the inner product value of the positive sample of the preset alignment model, is the sum of the inner product values ​​of the positive samples of the preset alignment model and all negative samples, Ι is the vector of the first image, T is the vector of the text, i is the vector of the i-th row of the first image matrix, k is the vector of the k-th row of the text matrix, and τ is a hyperparameter.

[0085] Step S40: input the alignment result into the decoder of the preset transformer model, output a plurality of keywords corresponding to the document to be processed, concatenate all the keywords in order, and input a summary of the document to be processed.

[0086] Specifically, step S40 includes:

[0087] A1. According to the arrangement order of each sentence of the text, at a first moment, input the alignment result into the decoder of the preset transformation model to generate a summary, thereby generating a first keyword;

[0088] A2. At a second moment, inputting the word vector of the first keyword and the alignment result into the decoder of the preset transformation model to generate a summary, thereby generating a second keyword;

[0089] A3. Repeat A1-A2 until the keywords of each sentence in the text are obtained.

[0090] The alignment result is input into the decoder of the preset transformer model for summary generation. The decoder will gradually generate summary words according to the arrangement order of each sentence in the text. For example, after the first keyword is generated at the first moment (t0), the word vector of the first keyword is used as input and then input into the decoder together with the alignment result. The second keyword is generated at the second moment (t1), and so on to obtain all keywords. Finally, all keywords are concatenated in order to obtain the summary sentence of the document to be processed.

[0091] Reference Figure 2, which is a schematic diagram of the functional modules of the text summary generating device 100 of the present invention.

[0092] The text summary generating device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the text summary generating device 100 may include an acquisition module 110, a word segmentation module 120, an output module 130 and a generation module 140. The module described in the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0093] In this embodiment, the functions of each module / unit are as follows:

[0094] Acquisition module 110: used for dividing a first image contained in a document to be processed into a plurality of second images, inputting the plurality of second images into a preset transformer model for patch embedding, and outputting image features of each second image;

[0095] The word segmentation module 120 is used to segment the text of the document to be processed according to a preset word segmentation algorithm to obtain multiple word segments, input each word segment into a preset transformer model for tag embedding, and output a word vector feature of each word segment;

[0096] Output module 130: used for inputting the image features of each second picture and the word vector features of each word segment into a preset alignment model for alignment processing, and outputting the alignment result;

[0097] The generation module 140 is used to input the alignment result into the decoder of the preset transformer model, output multiple keywords corresponding to the document to be processed, concatenate all the keywords in order, and input the summary of the document to be processed.

[0098] In one embodiment, inputting the image features of each second picture and the word vector features of each word segment into a preset alignment model for alignment processing, and outputting the alignment result, comprises:

[0099] Input the image features of each of the second pictures and the word vector features of each of the segmented words into a preset alignment model to perform attention weight allocation, and output a first initial value of each of the second pictures and a second initial value of each of the segmented words;

[0100] Performing a pooling operation on the first initial value and the second initial value respectively to obtain a vector of each of the second pictures and a vector of each of the word segments;

[0101] According to the vector of each of the second images and the vector of each of the word segments, a preset number of circular alignments are performed on the image features of each of the second images and the word vector features of each of the word segments, and an alignment result is output.

[0102] In one embodiment, performing a preset number of loop alignments on the image features of each second picture and the word vector features of each word segmentation according to the vector of each second picture and the vector of each word segmentation, and outputting the alignment result, includes:

[0103] Performing a pooling operation on all the image features of the second image to obtain a vector of the first image, and performing a pooling operation on the word vector features of all the word segments to obtain a vector of the text;

[0104] According to a preset first loss value calculation formula, the vector of the first image and the vector of each of the second images are compared and trained to obtain a first loss value, and according to a preset second loss value calculation formula, the vector of the text and the vector of each of the word segments are compared and trained to obtain a second loss value;

[0105] According to the first loss value and the second loss value, the parameters of the preset alignment model in the back propagation are adjusted, and the image features of each of the second pictures and the word vector features of each of the word segments are aligned for a preset number of times, and the alignment result is output.

[0106] In one embodiment, the inputting the image features of each second picture and the word vector features of each word segment into a preset alignment model for attention weight allocation, and outputting the first initial value of each second picture and the second initial value of each word segment, comprises:

[0107] According to the attention mechanism of the preset alignment model, the image features of each of the second pictures and the word vector features of each of the word segments are assigned attention weights and similarity is calculated to obtain a first information vector corresponding to the image features of each of the second pictures, and output a second information vector corresponding to the word vector features of each of the word segments;

[0108] Input a first activation function and a second activation function, concatenate the first information vector and the second information vector, perform a preset number of alignment operations, and output a first initial value of each of the second images and a second initial value of each of the segmented words.

[0109] In one embodiment, the preset first loss value calculation formula includes:

[0110]

[0111] in, is the first loss value, K is the number of cycles of the preset alignment model, L is the number of samples sampled during the training of the preset alignment model, is the inner product value of the positive sample of the preset alignment model, is the sum of the inner product values ​​of the positive samples of the preset alignment model and all negative samples, Ι is the vector of the first image, T is the vector of the text, i is the vector of the i-th row of the first image matrix, k is the vector of the k-th row of the text matrix, and τ is a hyperparameter.

[0112] In one embodiment, the preset second loss value calculation formula includes:

[0113]

[0114] in, is the second loss value, K is the number of cycles of the preset alignment model, L is the number of samples sampled during the training of the preset alignment model, is the inner product value of the positive sample of the preset alignment model, is the sum of the inner product values ​​of the positive samples of the preset alignment model and all negative samples, Ι is the vector of the first image, T is the vector of the text, i is the vector of the i-th row of the first image matrix, k is the vector of the k-th row of the text matrix, and τ is a hyperparameter.

[0115] In one embodiment, the step of inputting the alignment result into a decoder of the preset transformer model and outputting a plurality of keywords corresponding to the document to be processed includes:

[0116] A1. According to the arrangement order of each sentence of the text, at a first moment, input the alignment result into the decoder of the preset transformation model to generate a summary, thereby generating a first keyword;

[0117] A2. At a second moment, inputting the word vector of the first keyword and the alignment result into the decoder of the preset transformation model to generate a summary, thereby generating a second keyword;

[0118] A3. Repeat A1-A2 until the keywords of each sentence in the text are obtained.

[0119] Reference Figure 3 FIG. 1 is a schematic diagram of a preferred embodiment of an electronic device 1 of the present invention.

[0120] The electronic device 1 includes but is not limited to: a memory 11, a processor 12, a display 13 and a network interface 14. The electronic device 1 is connected to a network through the network interface 14 to obtain raw data. The network may be a wireless or wired network such as an intranet, the Internet, a global system of mobile communication (GSM), a wideband code division multiple access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, a call network, etc.

[0121] Among them, the memory 11 includes at least one type of readable medium, and the readable medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a hard disk or memory of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped with the electronic device 1. Of course, the memory 11 can also include both the internal storage unit of the electronic device 1 and its external storage device. In this embodiment, the memory 11 is generally used to store the operating system and various application software installed on the electronic device 1, such as the program code of the text summary generation 10, etc. In addition, the memory 11 can also be used to temporarily store various types of data that have been output or are to be output.

[0122] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the program code of the text summary generation 10.

[0123] The display 13 may be referred to as a display screen or a display unit. In some embodiments, the display 13 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an organic light-emitting diode (OLED) touch device. The display 13 is used to display information processed in the electronic device 1 and to display a visual working interface, such as displaying the results of data statistics.

[0124] The network interface 14 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and the network interface 14 is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0125] Figure 3 Only the electronic device 1 having components 11 - 14 and the text summary generation 10 is shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0126] Optionally, the electronic device 1 may further include a user interface, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an organic light-emitting diode (Organic Light-Emitting Diode, OLED) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0127] The electronic device 1 may further include a radio frequency (RF) circuit, a sensor, an audio circuit, etc., which will not be described in detail here.

[0128] In the above embodiment, the processor 12 may implement the following steps when executing the text summary generation 10 stored in the memory 11:

[0129] Splitting a first image contained in a document to be processed into a plurality of second images, inputting the plurality of second images into a preset transformer model for patch embedding, and outputting image features of each second image;

[0130] Segmenting the text of the document to be processed according to a preset segmentation algorithm to obtain multiple segmentations, inputting each segmentation into a preset transformer model for tag embedding, and outputting a word vector feature of each segmentation;

[0131] Inputting the image features of each of the second pictures and the word vector features of each of the word segments into a preset alignment model for alignment processing, and outputting the alignment result;

[0132] The alignment result is input into the decoder of the preset transformer model, a plurality of keywords corresponding to the document to be processed are output, all the keywords are concatenated in sequence, and a summary of the document to be processed is input.

[0133] The storage device may be the memory 11 of the electronic device 1 , or may be another storage device that is communicatively connected to the electronic device 1 .

[0134] For a detailed description of the above steps, please refer to the above Figure 2 Functional module diagram of the embodiment of the text summary generation device 100 and Figure 1 Description of a flowchart of an embodiment of a method for generating a text summary.

[0135] In addition, an embodiment of the present invention further proposes a computer-readable medium, which may be non-volatile or volatile. The computer-readable medium may be any one or any combination of a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, etc. The computer-readable medium includes a data storage area and a program storage area, the data storage area stores data created according to the use of the blockchain node, and the program storage area stores a text summary generation 10, and the text summary generation 10 is executed by the processor to implement the following operations:

[0136] Splitting a first image contained in a document to be processed into a plurality of second images, inputting the plurality of second images into a preset transformer model for patch embedding, and outputting image features of each second image;

[0137] Segmenting the text of the document to be processed according to a preset segmentation algorithm to obtain multiple segmentations, inputting each segmentation into a preset transformer model for tag embedding, and outputting a word vector feature of each segmentation;

[0138] Inputting the image features of each of the second pictures and the word vector features of each of the word segments into a preset alignment model for alignment processing, and outputting the alignment result;

[0139] The alignment result is input into the decoder of the preset transformer model, a plurality of keywords corresponding to the document to be processed are output, all the keywords are concatenated in sequence, and a summary of the document to be processed is input.

[0140] The specific implementation of the computer-readable medium of the present invention is substantially the same as the specific implementation of the above-mentioned text summary generation method, and will not be described in detail here.

[0141] In another embodiment, the text summary generation method provided by the present invention can further ensure the privacy and security of all the above-mentioned data, and all the above-mentioned data can also be stored in a blockchain node. For example, image features and word vector features can all be stored in the blockchain node.

[0142] It should be noted that the blockchain referred to in the present invention is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, platform product service layer, and application service layer.

[0143] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, an electronic device, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0145] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A text summary generation method, characterized in that: The method comprises: Splitting a first image contained in a document to be processed into a plurality of second images, inputting the plurality of second images into a preset transformer model for patch embedding, and outputting image features of each second image; Segmenting the text of the document to be processed according to a preset segmentation algorithm to obtain multiple segmentations, inputting each segmentation into a preset transformer model for tag embedding, and outputting a word vector feature of each segmentation; Inputting the image features of each of the second pictures and the word vector features of each of the word segments into a preset alignment model for alignment processing, and outputting the alignment result; Input the alignment result into the decoder of the preset transformer model, output multiple keywords corresponding to the document to be processed, concatenate all the keywords in order, and input the summary of the document to be processed; Wherein, the inputting the image features of each second picture and the word vector features of each word segment into a preset alignment model for alignment processing, and outputting the alignment result, includes: inputting the image features of each second picture and the word vector features of each word segment into a preset alignment model for attention weight allocation, and outputting the first initial value of each second picture and the second initial value of each word segment; performing pooling operations on the first initial value and the second initial value respectively to obtain the vector of each second picture and the vector of each word segment; performing a preset number of cyclic alignments on the image features of each second picture and the word vector features of each word segment according to the vector of each second picture and the vector of each word segment, and outputting the alignment result; According to the vector of each second picture and the vector of each word segment, the image features of each second picture and the word vector features of each word segment are aligned for a preset number of times, and the alignment result is output, including: pooling the image features of all second pictures to obtain the vector of the first picture, and pooling the word vector features of all each word segment to obtain the vector of the text; according to a preset first loss value calculation formula, the vector of the first picture and the vector of each second picture are compared and trained to obtain a first loss value, and according to a preset second loss value calculation formula, the vector of the text and the vector of each word segment are compared and trained to obtain a second loss value; according to the first loss value and the second loss value, the parameters of the preset alignment model in the back propagation are adjusted, and the image features of each second picture and the word vector features of each word segment are aligned for a preset number of times, and the alignment result is output; The method of inputting the image features of each second picture and the word vector features of each word segment into a preset alignment model for attention weight allocation, and outputting the first initial value of each second picture and the second initial value of each word segment, includes: according to the attention mechanism of the preset alignment model, allocating attention weights and calculating similarity for the image features of each second picture and the word vector features of each word segment, obtaining the first information vector corresponding to the image features of each second picture, and outputting the second information vector corresponding to the word vector features of each word segment; inputting the first activation function and the second activation function, splicing the first information vector and the second information vector and performing a preset number of alignment operations, and outputting the first initial value of each second picture and the second initial value of each word segment.

2. The text summary generation method according to claim 1, characterized in that: The parameters of the preset first loss value calculation formula include: The number of cycles of the preset alignment model, the number of samples sampled during the training of the preset alignment model, the inner product value of the positive samples of the preset alignment model, the sum of the inner product values ​​of the positive samples of the preset alignment model and all negative samples, the vector of the first image, the vector of the text, the first image matrix The vector of rows, the first A vector of rows, a hyperparameter.

3. The text summary generation method according to claim 2, characterized in that: The parameters of the preset second loss value calculation formula include: The number of cycles of the preset alignment model, the number of samples sampled during the training of the preset alignment model, the inner product value of the positive samples of the preset alignment model, the sum of the inner product values ​​of the positive samples of the preset alignment model and all negative samples, the vector of the first image, the vector of the text, the first image matrix The vector of rows, the first A vector of rows, a hyperparameter.

4. The text summary generation method according to claim 1, characterized in that: The step of inputting the alignment result into a decoder of the preset transformer model and outputting a plurality of keywords corresponding to the document to be processed comprises: A1. According to the arrangement order of each sentence of the text, at a first moment, input the alignment result into the decoder of the preset transformer model to generate a summary, thereby generating a first keyword; A2. At a second moment, inputting the word vector of the first keyword and the alignment result into the decoder of the preset transformer model to generate a summary, thereby generating a second keyword; A3. Repeat A1-A2 until the keywords of each sentence in the text are obtained.

5. A text summary generation device, used to implement the text summary generation method according to any one of claims 1 to 4, characterized in that: The device comprises: An acquisition module: used for dividing a first image contained in a document to be processed into a plurality of second images, inputting the plurality of second images into a preset transformer model for patch embedding, and outputting image features of each second image; Word segmentation module: used for segmenting the text of the document to be processed according to a preset word segmentation algorithm to obtain multiple word segments, inputting each word segment into a preset transformer model for tag embedding, and outputting word vector features of each word segment; Output module: used for inputting the image features of each second picture and the word vector features of each word segmentation into a preset alignment model for alignment processing, and outputting the alignment result; Generation module: used for inputting the alignment result into the decoder of the preset transformer model, outputting multiple keywords corresponding to the document to be processed, splicing all the keywords in order, and inputting the summary of the document to be processed.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a program executable by the at least one processor, and the program is executed by the at least one processor so that the at least one processor performs the text summary generating method according to any one of claims 1 to 4.

7. A computer-readable medium, characterized in that The computer-readable medium stores text summary generation, and when the text summary generation is executed by a processor, the text summary generation method according to any one of claims 1 to 4 is implemented.

Citation Information

Patent Citations

  • Text abstract method and device, electronic equipment and storage medium

    CN111428025A

  • Automatic question answering method and device based on image-text chat record, equipment and medium

    CN113821602A