Method, apparatus and electronic device for generating abstract

Abstract

The application provides a summary generation method and device and electronic equipment, and belongs to the technical field of computer application. The summary generation method comprises the following steps: obtaining a summary generation request, wherein the summary generation request comprises a description text and an image; inputting the description text into a preset first encoder to generate a first hidden layer sequence corresponding to the description text; inputting the image into a preset second encoder to generate a feature vector corresponding to the image; correcting the first hidden layer sequence according to the feature vector to generate a second hidden layer sequence corresponding to the description text; and inputting the second hidden layer sequence into a preset decoder to generate a target summary corresponding to the description text. Thus, by using the summary generation method, the feature vector corresponding to the image is used to correct the hidden layer sequence corresponding to the description text, so that the key information in the description text information is mined through the image information, and the accuracy of the summary generation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer application, and in particular to a summary generation method and device and electronic equipment. BACKGROUND

[0002] The input of a multi-modal automatic summarization task is a detailed text and an image, and the output is a short summary. In the related art, the text and the image can be encoded respectively to generate a text semantic vector and an image semantic vector, and a short summary can be generated based on a multi-modal attention mechanism and the text semantic vector and the image semantic vector.

[0003] However, since the amount of information contained in the text is much larger than that of the image, it is unreasonable to process the text and the image equally, which can easily lead to poor accuracy of the generated summary. SUMMARY

[0004] The summary generation method and device, electronic equipment and storage medium provided by the present application can solve the problem that, in the related art, since the amount of information contained in the text is much larger than that of the image, the method of processing the text and the image equally to generate a short summary can easily lead to poor accuracy of the generated summary.

[0005] The summary generation method provided by an embodiment of the present application includes: obtaining a summary generation request, wherein the summary generation request includes a description text and an image; inputting the description text into a preset first encoder to generate a first hidden layer sequence corresponding to the description text; inputting the image into a preset second encoder to generate a feature vector corresponding to the image; correcting the first hidden layer sequence according to the feature vector to generate a second hidden layer sequence corresponding to the description text; and inputting the second hidden layer sequence into a preset decoder to generate a target summary corresponding to the description text.

[0006] The summary generation device provided by another embodiment of the present application includes: an obtaining module configured to obtain a summary generation request, wherein the summary generation request includes a description text and an image; a first generation module configured to input the description text into a preset first encoder to generate a first hidden layer sequence corresponding to the description text; a second generation module configured to input the image into a preset second encoder to generate a feature vector corresponding to the image; a correction module configured to correct the first hidden layer sequence according to the feature vector to generate a second hidden layer sequence corresponding to the description text; and a third generation module configured to input the second hidden layer sequence into a preset decoder to generate a target summary corresponding to the description text.

[0007] In another aspect, the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method for generating an abstract when executing the program.

[0008] In another aspect, the computer readable storage medium stores a computer program, and the program is executed by a processor to implement the method for generating an abstract.

[0009] The method and device for generating an abstract, the electronic device, and the computer readable storage medium provided by the embodiments of the present application are provided. When the abstract generation request is obtained, the description text in the abstract generation request is input into a preset first encoder to generate a first hidden layer sequence corresponding to the description text, and an image is input into a preset second encoder to generate a feature vector corresponding to the image. Then, the first hidden layer sequence is corrected according to the feature vector to generate a second hidden layer sequence corresponding to the description text, and then the second hidden layer sequence is input into a preset decoder to generate a target abstract corresponding to the description text. In this way, the hidden layer sequence corresponding to the description text is corrected by using the feature vector corresponding to the image to mine key information in the description text information through image information, thereby improving the accuracy of abstract generation.

[0010] Additional aspects and advantages of the present application will be made apparent from the following description, which, taken together with the accompanying drawings, will illustrate some embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 A flowchart of a method for generating an abstract provided by the embodiments of the present application is shown in FIG. 1;

[0013] Figure 2 An image corresponding to a commodity is shown in FIG. 2;

[0014] Figure 3 A schematic diagram of an abstract generation model provided by the embodiments of the present application is shown in FIG. 3;

[0015] Figure 4 A schematic diagram of an encoding process of a Bi-LSTM encoder is shown in FIG. 4;

[0016] Figure 5 A flowchart of another method for generating an abstract provided by the embodiments of the present application is shown in FIG. 5;

[0017] Figure 6 A structural schematic diagram of an abstract generation device provided by the embodiments of the present application is shown in FIG. 6;

[0018] Figure 7 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] Embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar reference numerals represent the same or similar elements throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0020] Embodiments of the present application aim at the problem in the related art that the method of processing text and image equally to generate a short summary easily leads to poor accuracy of summary generation, because the information contained in the text is much more than the image.

[0021] The summary generation method provided by the embodiments of the present application, by inputting the description text in the summary generation request into a preset first encoder when the summary generation request is obtained, generating a first hidden layer sequence corresponding to the description text, and inputting the image into a preset second encoder, generating a feature vector corresponding to the image, then correcting the first hidden layer sequence according to the feature vector to generate a second hidden layer sequence corresponding to the description text, and further inputting the second hidden layer sequence into a preset decoder to generate a target summary corresponding to the description text. Thus, by using the feature vector corresponding to the image to correct the hidden layer sequence corresponding to the description text, the key information in the description text information is mined through image information, thereby improving the accuracy of summary generation.

[0022] The summary generation method, device, electronic device, storage medium and computer program provided by the present application are described in detail below with reference to the drawings.

[0023] Figure 1 A flowchart of a summary generation method provided by an embodiment of the present application is shown.

[0024] As shown in Figure 1 The summary generation method comprises the following steps:

[0025] Step 101, obtaining a summary generation request, wherein the summary generation request comprises a description text and an image.

[0026] It should be noted that the summary generation method of the embodiments of the present application can be executed by the summary generation device of the embodiments of the present application. The summary generation device of the embodiments of the present application can be configured in any electronic device to execute the summary generation method of the embodiments of the present application.

[0027] For example, the generation method of the summary of the embodiment of the present application can be configured in the server corresponding to any e-commerce service to generate a summary of a commodity according to the description text and the commodity image corresponding to the commodity.

[0028] Among them, the image can be any type of image, such as a commodity image; the description text can be the description text corresponding to the image.

[0029] For example, the image is the image of commodity A, and the description text can be the description information of commodity A; assuming that commodity A is a mobile phone, the description text can be "X brand Y series Z model mobile phone, 5G dual-mode full-network access, large-size super-curved ring screen allows screen display to present ring visual, Kirin 990 5G flagship SoC chip powerful computing power, peak speed, rear camera 4000 million pixel movie camera + 4000 million pixel super-sensory camera + 800 million pixel long-focus camera + 3D depth camera, super-high-definition night shooting, 7680 frame super-high-speed photography, 4500 mAh strong endurance, double super-fast charging", and the image of commodity A is an image describing the configuration parameters of commodity A, such as Figure 2 .

[0030] As a possible implementation, the server can obtain the image and the description text corresponding to the image from the database or the network according to the pre-set rule, and generate a summary generation request according to the obtained image and the corresponding description text to generate a summary corresponding to the description text and the image. For example, when the server is the server corresponding to the e-commerce application, the pre-set rule is "generate a summary of a new commodity when a new commodity is obtained", and the server can obtain the commodity image and the description text corresponding to the new commodity when it is determined that a new commodity is added to the database, and determine that the summary generation request is obtained.

[0031] As another possible implementation, the server can also determine that the summary generation request is obtained when the image and the description text corresponding to the image sent by the user through the client are obtained.

[0032] Step 102, input the description text into the preset first encoder to generate a first hidden layer sequence corresponding to the description text.

[0033] Among them, the preset first encoder can be a pre-trained deep learning model that can generate a semantic vector corresponding to the text. In actual use, a suitable encoder can be selected according to actual needs and specific application scenarios, and the present application does not limit this. For example, the preset first encoder can be a pre-trained Bi-directional Long Short-Term Memory (Bi-LSTM) encoder.

[0034] The first hidden layer sequence can be a vector that can represent the semantics of the description text. In actual use, the first hidden layer sequence generated by the preset first encoder can be different when the preset first encoder is different, and the present application does not limit this.

[0035] In the embodiment of the present application, after obtaining the summary generation request, the description text and the image can be encoded respectively to generate vector representations of the description text and the image. For example, as shown in Figure 3 FIG. 1 is a schematic diagram of a summary generation model according to an embodiment of the present application. The description text can be input into a preset first encoder in the model to enable the preset first encoder to encode the description text and generate a first hidden layer sequence h corresponding to the description text. Figure 3

[0036] As a possible implementation, the description text usually includes multiple tokens, so the description text can be tokenized, and each token included in the description text can be input into the preset first encoder in turn to generate a first hidden layer sequence corresponding to the description text. That is, in a possible implementation of the embodiment of the present application, the step 102 can include:

[0037] Each first token can be input into the preset first encoder to generate a first hidden layer vector corresponding to each first token.

[0038] The first hidden layer vectors corresponding to the multiple first tokens can be fused to generate the first hidden layer sequence.

[0039] The first token refers to each token included in the description text.

[0040] In the embodiment of the present application, the description text can be tokenized by using a tokenization algorithm before being input into the preset first encoder to obtain multiple first tokens included in the description text, and then each first token can be input into the preset first encoder to generate a first hidden layer vector corresponding to each first token. The first hidden layer vector corresponding to the first token can represent the semantics, context features, etc. of the first token. Then, the first hidden layer vectors corresponding to the multiple first tokens can be fused to generate a first hidden layer sequence h corresponding to the description text.

[0041] Optionally, the fusion of the multiple first hidden layer vectors can be concatenation of the multiple first hidden layer vectors, or summation of the multiple first hidden layer vectors, or mean value of the multiple first hidden layer vectors, etc., and the present application does not limit this.

[0042] For example, the preset first encoder is a BERT model as shown in Figure 4 ​The Bi-LSTM encoder, the first i first token x i The corresponding first hidden layer vector h i may be represented by formula (1). As can be seen, the first i first token x i The corresponding first hidden layer vector h i is related to the first i first token x i and the first i-1 first token x i-1 The corresponding first hidden layer vector (the output of the previous time of the Bi-LSTM encoder).

[0043] h i =BILSTM(x i ,h i-1 ) (1)

[0044] Where h i is the first i first token x i corresponding to the first hidden layer vector of the description text, x i is the first i first token of the description text, and h i-1 is the first i-1 first token x i-1 corresponding to the first hidden layer vector of the description text.

[0045] As Figure 4 shown, it is a schematic diagram of the encoding process of a Bi-LSTM encoder, wherein LSTM L is a forward LSTM, and LSTM R is a backward LSTM. Assuming that the description text is "I love China", the first token included in the description text is "I", "love", and "China", respectively, "I", "love", and "China" are input into the forward LSTM L in turn, and three hidden vectors h L0 , h L1 , and h L2 may be output at three time points, respectively, and "China", "love", and "I" are input into the backward LSTM R in turn, and three hidden vectors h R0 , h R1 , and h R2 may be output at three time points, respectively; h L0 and h R0 may then be spliced to generate the first hidden layer vector h0 corresponding to the first token "I", h L1 and h R1 may be spliced to generate the first hidden layer vector h1 corresponding to the first token "love", and h L2 and h R2The first hidden layer vectors h0, h1, h2, …, h L0 , corresponding to the first segmented words in the description text are determined. R0 , h1 = [h L1 , h R1 ], h2 = [h L2 , h R2 ].

[0046] It should be noted that the above examples are only exemplary and should not be considered as limiting the present application. In actual use, the description text actually to be processed can be input into the Bi-LSTM encoder as shown in Figure 4 to generate the first hidden layer vectors corresponding to each first segmented word in the actual description text. For example, in the above example, the description text: “X brand Y series Z model mobile phone, 5G dual-mode full-network access, large-size super-curved ring screen allows screen display to present ring visual, Kirin 990 5G flagship SoC chip powerful computing power, peak speed, rear camera 4000 million pixel movie camera + 4000 million pixel super-sensory camera + 800 million pixel long-focus camera + 3D depth camera, ultra-high-definition night shooting, 7680 frame super-high-speed photography, 4500 mAh strong endurance, double super-fast charging”, after being segmented to generate 67 first segmented words, the first hidden layer vectors corresponding to each first segmented word can be h0, h1, h2, …, h 66 .

[0047] As a possible implementation, the first hidden layer vectors corresponding to each first segmented word in the description text are determined and spliced to generate the first hidden layer sequence corresponding to the description text.

[0048] For example, in the above example, the first hidden layer vectors h0, h1, h2, …, h 66 corresponding to each segmented word in the description text are determined, and the first hidden layer sequence h corresponding to the description text can be [h0, h1, h2, …, h 66 ].

[0049] Step 103, inputting the image into a preset second encoder to generate a feature vector corresponding to the image.

[0050] The preset second encoder can be a deep learning model pre-trained to generate a semantic vector corresponding to the image. In actual use, a suitable encoder can be selected according to actual needs and specific application scenarios, and the present application embodiments do not limit this. For example, the preset second encoder can be a pre-trained convolutional neural network (CNN) model, such as VGG16, VGG19, etc.

[0051] The feature vector corresponding to the image can be a vector that can represent the content and / or semantics of the image. In actual use, the generated feature vector can be different when the preset second encoder is different, and the embodiments of the present application do not limit this.

[0052] It should be noted that the feature vector corresponding to the image can be a global feature vector that can represent the global features of the image, or a local feature vector that can represent the local key features of the image. In actual use, the type of the feature vector corresponding to the image can be determined according to actual needs and specific application scenarios, and the embodiments of the present application do not limit this.

[0053] In the embodiments of the present application, after obtaining the summary generation request, the description text and the image can be encoded respectively to generate vector representations of the description text and the image. Therefore, the image can be input into the preset second encoder shown in FIG. 10 to enable the preset second encoder to encode the image to generate a feature vector v corresponding to the image. Figure 3

[0054] Step 104: correcting the first hidden layer sequence according to the feature vector to generate a second hidden layer sequence corresponding to the description text.

[0055] In the embodiments of the present application, since the description text usually contains relatively rich information, and the image usually only contains relatively less description information, directly decoding the first hidden layer sequence corresponding to the description text and the feature vector corresponding to the image to generate a summary cannot well balance the importance of the text and the image, thereby resulting in poor quality of the generated summary. However, since the description text contains a large amount of information, it also contains a large amount of redundant information; and the image can directly describe the features of the object, so the image can contain key description information. Therefore, in the embodiments of the present application, the feature vector corresponding to the image can be used to correct the first hidden layer vector corresponding to the description text to generate a second hidden layer sequence that can not only represent the semantics of the description text but also represent the key information of the description text, so as to filter the key information in the description text through the image and improve the quality of the summary generation.

[0056] As a possible implementation manner, when the description text includes a plurality of first tokens, the feature vector corresponding to the image can be used to correct the first hidden layer vector corresponding to each token respectively to determine the importance of each first token, thereby improving the quality of the summary generation. That is, in one possible implementation manner of the embodiments of the present application, the above step 104 can include:

[0057] correcting each first hidden layer vector according to the feature vector to generate a second hidden layer vector corresponding to each first token; ​

[0058] The second hidden layer vectors corresponding to the plurality of first segmented words are fused to generate a second hidden layer sequence.

[0059] In the embodiments of the present application, the first hidden layer vector corresponding to each first segmented word in the description text can be corrected by using the feature vector corresponding to the image to generate the second hidden layer vector corresponding to each first segmented word, so that the second hidden layer vector corresponding to the first segmented word can not only represent the semantic, context feature, etc. of the first segmented word in the description text, but also represent the importance of the first segmented word in the description text. Then, the second hidden layer vector corresponding to each first segmented word is fused to generate the second hidden layer sequence corresponding to the description text, so that the second hidden layer sequence can not only represent the semantic of each first segmented word in the description text, but also fuse the importance of each first segmented word, and then realize the key information screening of the description text through the importance of each segmented word.

[0060] It should be noted that the fusion manner of the second hidden layer vector can be the same as the fusion manner of the first hidden layer vector, which will not be described here. For example, the first hidden layer sequence is generated by splicing the first hidden layer vectors corresponding to the plurality of first segmented words, and the second hidden layer sequence can also be generated by splicing the second hidden layer sequences corresponding to the plurality of first segmented words.

[0061] Further, in order to improve the accuracy of the key information screening of the description text, a weight matrix for extracting the importance of the first segmented word can be pre-trained to determine the correction of the first hidden layer vector of each first segmented word by the weight matrix and the feature vector corresponding to the image, so as to introduce the importance information in the vector representation of the first segmented word. That is, in a possible implementation manner of the embodiments of the present application, the above-mentioned correction of each first hidden layer vector according to the feature vector to generate the second hidden layer vector corresponding to each first segmented word comprises:

[0062] determining the mean of the plurality of first hidden layer vectors corresponding to the plurality of first segmented words;

[0063] obtaining a first weight matrix corresponding to the first hidden layer vector, a second weight matrix corresponding to the mean of the first hidden layer vector, and a third weight matrix corresponding to the feature vector;

[0064] correcting each first hidden layer vector according to the first weight matrix, the second weight matrix, the third weight matrix, the mean of the first hidden layer vector, and the feature vector to generate the second hidden layer vector corresponding to each first segmented word.

[0065] In the embodiment of the present application, the first weight matrix corresponding to the first hidden layer vector, the second weight matrix corresponding to the mean of the first hidden layer vectors, and the third weight matrix corresponding to the feature vector of the image can be pre-trained, so as to fuse the first hidden layer vector, the mean of the plurality of first hidden layer vectors, and the feature vector of the image through the first weight matrix, the second weight matrix, and the third weight matrix, thereby generating the corrected first hidden layer vector, i.e., the second hidden layer vector fused with the first segmentation importance information. Optionally, the second hidden layer vector corresponding to the first segmentation can be determined through formula (2).

[0066]

[0067] wherein h' i is the second hidden layer vector corresponding to the i th first segmentation in the description text, h i is the first hidden layer vector corresponding to the i th first segmentation in the description text, σ is a Sigmoid function, N is the number of the first segmentations included in the description text, i is the serial number of the first segmentation in the description text, v is the feature vector of the image, W1 is the first weight matrix, W2 is the second weight matrix, and W3 is the third weight matrix. i i wherein h' i is the second hidden layer vector corresponding to the i th first segmentation in the description text, h i is the first hidden layer vector corresponding to the i th first segmentation in the description text, σ is a Sigmoid function, N is the number of the first segmentations included in the description text, i is the serial number of the first segmentation in the description text, v is the feature vector of the image, W1 is the first weight matrix, W2 is the second weight matrix, and W3 is the third weight matrix.

[0068] In step 105, the second hidden layer sequence is input into a preset decoder to generate a target summary corresponding to the description text.

[0069] In the embodiment of the present application, after the second hidden layer sequence corresponding to the description text is determined, the second hidden layer sequence can be decoded by using a preset decoder to generate a target summary corresponding to the description text. Since the importance information of each first segmentation in the description text is fused in the second hidden layer sequence, the accuracy of the summary generation is improved.

[0070] For example, for the description text of the product A in the above example, which is “X brand Y series Z model mobile phone, 5G dual-mode full-network access, large-size super-curved ring screen makes the screen display present ring visual, Kirin 990 5G flagship SoC chip powerful computing power, peak speed, rear camera is 4000 million pixel movie camera + 4000 million pixel super-sensory camera + 800 million pixel long-focus camera + 3D deep-sensing camera, super-high-definition night shooting, 7680 frame super-high-speed photography, 4500 mAh strong endurance, double super-fast charging”, the corresponding image is as shown in FIG. 2, and the generated target summary can be “X brand Y series Z model 5G Kirin 990 dual 4000 million Leica four cameras”. Figure 2

[0071] ​​The abstract generation method provided in the embodiments of the present application comprises the following steps: when an abstract generation request is obtained, inputting a description text in the abstract generation request into a preset first encoder to generate a first hidden layer sequence corresponding to the description text, and inputting an image into a preset second encoder to generate a feature vector corresponding to the image; then, the first hidden layer sequence is corrected according to the feature vector to generate a second hidden layer sequence corresponding to the description text; and then, the second hidden layer sequence is input into a preset decoder to generate a target abstract corresponding to the description text. In this way, the hidden layer sequence corresponding to the description text is corrected by using the feature vector corresponding to the image to mine key information in the description text information through image information, thereby improving the accuracy of abstract generation.

[0072] In a possible implementation form of the present application, the second hidden layer sequence corresponding to the description text can be decoded by using an LSTM decoder, and a target abstract is generated by selecting a word segmentation matched with the decoding result from a preset dictionary, so as to further improve the accuracy of abstract generation.

[0073] The abstract generation method provided in the embodiments of the present application will be further described below. Figure 5

[0074] Figure 5 Another flowchart of the abstract generation method provided in the embodiments of the present application is shown in the figure.

[0075] As shown in the figure, the abstract generation method comprises the following steps: Figure 5

[0076] Step 201: obtaining an abstract generation request, wherein the abstract generation request comprises a description text and an image.

[0077] Step 202: inputting the description text into a preset first encoder to generate a first hidden layer sequence corresponding to the description text.

[0078] Step 203: inputting the image into a preset second encoder to generate a feature vector corresponding to the image.

[0079] Step 204: correcting the first hidden layer sequence according to the feature vector to generate a second hidden layer sequence corresponding to the description text.

[0080] The specific implementation process and principle of the above step 204 can be referred to the detailed description of the above embodiments, which will not be described here again.

[0081] Step 205: determining an attention score of each first word segmentation at time t.

[0082] ​​It should be noted that the preset decoder of the embodiment of the present application can be an LSTM decoder, for example, can be a unidirectional LSTM decoder. In actual use, a suitable decoder can be selected according to actual needs and specific application scenarios, and the embodiment of the present application does not limit this. The following will be described in detail taking the preset decoder as a unidirectional LSTM decoder.

[0083] Wherein, the target summary can include multiple target summary words; t time, refers to the time when the preset decoder generates the tth target summary word in the target summary.

[0084] In the embodiment of the present application, when decoding the second hidden layer sequence corresponding to the description text, the attention mechanism can be introduced to further improve the accuracy of the summary generation. Optionally, the preset decoder can generate the target summary in the unit of word segmentation, so that the preset decoder can generate each target summary word in the target summary in turn, and determine the attention score of each first word segmentation when generating the summary word to introduce the attention mechanism.

[0085] As a possible implementation, when the tth target summary word is generated, the attention score of each first word segmentation at t time can be determined according to the second hidden layer vector corresponding to the first word segmentation and the (t-1)th target hidden layer sequence generated when the preset decoder decodes the second hidden layer sequence at (t-1) time (i.e. the time when the (t-1)th target summary word is generated). That is, in a possible implementation of the embodiment of the present application, the above step 205 can include:

[0086] According to the second hidden layer vector corresponding to each first word segmentation and the (t-1)th target hidden layer sequence, the attention score of each first word segmentation at t time is determined.

[0087] Wherein, the (t-1)th target hidden layer sequence refers to the hidden layer sequence generated at (t-1) time when the preset decoder decodes the second hidden layer sequence, i.e. the vector representation corresponding to the (t-1)th target summary word. It should be noted that the (t-1)th target hidden layer sequence can represent the semantic information of the (t-1)th target summary word.

[0088] As a possible implementation, the attention score of each first word segmentation at t time can be determined by formula (3) and formula (4).

[0089]

[0090] α t,i =softmax(e t,i ) (4)

[0091] Wherein, α t,ito describe the attention score of the i th first segmented word in the text at time t, e t,i to describe the intermediate variable of the i th first segmented word in the text at time t, h′ i to describe the second hidden layer vector corresponding to the i th first segmented word in the text, i is the serial number of the first segmented word, s t-1 to describe the (t-1) th target hidden layer sequence, u a , W a , V a , W b and V b are the preset model parameter matrix corresponding to the decoder, and are the model parameters trained in advance.

[0092] For example, the description text is “X brand Y series Z model mobile phone, 5G dual-mode full-network, large-size super-curved ring screen makes the screen display present a ring visual effect, Kirin 990 5G flagship SoC chip with powerful computing power and peak speed, rear camera with 40 million pixel movie camera, 40 million pixel super-sensory camera, 8 million pixel long-focus camera and 3D depth camera, ultra-high-definition night shooting, 7680 frame super-high-speed photography, 4500 mAh strong endurance, double super-fast charging”, and the first target summary word generated is “X brand”, then the attention score of the third first segmented word “Z model” in the description text at the time of generating the second target summary word is wherein h′3 is the second hidden layer vector corresponding to the third first segmented word “Z model”, and s1 is the target hidden layer sequence corresponding to the first target summary word “X brand” generated by the preset decoder.

[0093] Step 206, determining the context vector of the description text at time t according to the attention score of each first segmented word at time t and each second hidden layer vector.

[0094] wherein the context vector of the description text at time t can represent the context feature of the description text at time t.

[0095] In the embodiment of the present application, after determining the attention score of each first segmented word in the description text at time t, the attention score of the first segmented word at time t can be fused with each second hidden layer vector to generate the context vector of the description text at time t. Optionally, the context vector of the description text at time t can be determined by formula (5).

[0096] c t =∑ i α t,i h′ i (5)

[0097] wherein c t is the context vector of the description text at time t, and αt,i to describe the attention score of the i-th first segmented word in the text at the t time, h' i,t i to describe the second hidden layer vector corresponding to the i-th first segmented word in the text, i is the serial number of the first segmented word.

[0098] At step 207, the t-th target hidden layer sequence is generated according to the t-1-th target hidden layer sequence generated by the preset decoder at the t-1 time, the t-1-th target summary word, and the context vector describing the text at the t time.

[0099] In the embodiment of the present application, after the context vector describing the text at the t time is determined through the attention mechanism, the preset decoder can be used to continue decoding the t-1-th target hidden layer sequence, the t-1-th target summary word, and the context vector describing the text at the t time to determine the t-th target hidden layer sequence. The t-th target hidden layer sequence can represent the semantic information of the t-th target summary word.

[0100] As a possible implementation, the t-th target hidden layer sequence can be determined by formula (6).

[0101] s t = f dec (s t-1 ,y t-1 ,c t ) (6)

[0102] Where s t is the t-th target hidden layer sequence, f dec is a function corresponding to the preset decoder, s t-1 is the t-1-th target hidden layer sequence, y t-1 is the t-1-th target summary word that has been generated, and c t is the context vector describing the text at the t time.

[0103] For example, the text describes "X brand Y series Z model mobile phone, 5G dual-mode full-network, large-size super-curved ring screen makes the screen display present a ring visual, Kirin 990 5G flagship SoC chip powerful computing power, peak speed, rear camera 4000 million pixel movie camera + 4000 million pixel super-sensory camera + 800 million pixel long-focus camera + 3D depth camera, super-high-definition night shooting, 7680 frame super-high-speed photography, 4500 mAh strong endurance, double super-fast charging", and the first target summary word generated is "X brand". The preset decoder can generate the second target summary word according to the target hidden layer sequence corresponding to the first target summary word "X brand", the first target summary word "X brand", and the context vector c2 corresponding to the text at the time of generating the second target summary word.

[0104] In step 208, the tth target summary word is obtained from the preset dictionary according to the tth target hidden layer sequence and the context vector of the description text at the tth moment.

[0105] The preset dictionary can include a large number of segmented words and parameter matrices corresponding to each segmented word. It should be noted that the parameter matrix corresponding to the segmented word can be a pre-generated parameter, which can be used to represent semantic information of the segmented word.

[0106] In the embodiment of the present application, the correlation degree of the tth target hidden layer sequence and each segmented word in the preset dictionary can be determined according to the tth target hidden layer sequence and the context vector of the description text at the tth moment, and then the segmented word with the largest correlation degree with the tth target hidden layer sequence is determined as the tth target summary word corresponding to the tth target hidden layer sequence according to the correlation degree.

[0107] As a possible implementation, the probability of each segmented word being the tth target summary word can be determined according to the parameter matrix of each segmented word in the preset dictionary, and the tth target summary word is selected from the preset dictionary according to the probability. That is, in a possible implementation of the embodiment of the present application, step 208 can include:

[0108] Obtaining the first parameter matrix and the second parameter matrix corresponding to each second segmented word in the preset dictionary;

[0109] Determining the probability of each second segmented word being the tth target summary word according to the first parameter matrix and the second parameter matrix corresponding to each second segmented word, the tth target hidden layer sequence and the context vector of the description text at the tth moment;

[0110] Determining the second segmented word with the largest probability of being the tth target summary word as the tth target summary word.

[0111] The second segmented word refers to a segmented word included in the preset dictionary.

[0112] The first parameter matrix and the second parameter matrix corresponding to the second segmented word can be used to represent semantic information of the second segmented word, and can be pre-learned parameters.

[0113] As a possible implementation, the probability of each second segmented word being the tth target summary word can be determined by formula (7).

[0114] P vocab (w i )=softmax(W b,i s t +V b,i c t ) (7)

[0115] where Pvocab (w i ) is the probability that the i th second segmented word in the preset dictionary is the t th target abstract word, w i s is the i th second segmented word in the preset dictionary, s t c is the t th target hidden layer sequence, c t W is a context vector describing the text at time t, W b,i V is a first parameter matrix corresponding to the i th second segmented word in the preset dictionary, V b,i V is a second parameter matrix corresponding to the i th second segmented word in the preset dictionary.

[0116] In the embodiment of the present application, the greater the probability that the second segmented word is the t th target abstract word, the higher the quality of the target abstract generated according to the second segmented word, so the second segmented word with the greatest probability of being the t th target abstract word can be determined as the t th target abstract word.

[0117] Step 209, generating a target abstract according to the target abstract words generated at each time by the preset encoder.

[0118] In the embodiment of the present application, after the preset decoder decodes the second hidden layer sequence corresponding to the description text, it can output a plurality of target abstract words in turn, so that the plurality of target abstract words can be combined according to the output order of the plurality of target abstract words to generate a target abstract.

[0119] For example, the description text is "X brand Y series Z model mobile phone, 5G dual-mode full-network, large-size super-curved ring screen makes screen display present ring visual, Kirin 990 5G flagship SoC chip powerful computing power, peak speed, rear camera 4000 million pixel movie camera + 4000 million pixel super-sensory camera + 800 million pixel long-focus camera + 3D depth camera, super-high-definition night shooting, 7680 frame super-high-speed photography, 4500 mAh strong endurance, double super-fast charging", the corresponding image is as shown in Figure 2 The generated target abstract words are X brand, Y series, Z model, 5G, Kirin 990, double, 4000 million, Leica four cameras in turn, and the generated target abstract can be "X brand Y series Z model 5G Kirin 990 double 4000 million Leica four cameras".

[0120] The abstract generation method provided by the embodiment of the present application corrects the first hidden layer sequence corresponding to the description text according to the feature vector corresponding to the image, generates a second hidden layer sequence corresponding to the description text, determines the attention score of each first word segmentation at time t and the context vector of the description text at time t through the attention mechanism, then generates the tth target hidden layer sequence according to the (t-1)th target hidden layer sequence generated by the preset decoder at time t-1, the (t-1)th target abstract word, and the context vector of the description text at time t, and further acquires the tth target abstract word from the preset dictionary according to the tth target hidden layer sequence and the context vector of the description text at time t, and generates the target abstract according to the target abstract word generated by the preset encoder at each time. Therefore, the hidden layer sequence corresponding to the description text is corrected by using the feature vector corresponding to the image to mine the key information in the description text information through the image information, and the attention mechanism is introduced in the decoding process to select the word segmentation matching the decoding result from the preset dictionary to generate the target abstract, thereby further improving the accuracy of the abstract generation.

[0121] To implement the above embodiment, the present application further provides an abstract generation device.

[0122] Figure 6 The structure diagram of the abstract generation device provided by the embodiment of the present application.

[0123] As shown in Figure 6 The abstract generation device 30 comprises:

[0124] The acquisition module 31 is configured to acquire an abstract generation request, wherein the abstract generation request comprises a description text and an image.

[0125] The first generation module 32 is configured to input the description text into a preset first encoder to generate a first hidden layer sequence corresponding to the description text.

[0126] The second generation module 33 is configured to input the image into a preset second encoder to generate a feature vector corresponding to the image.

[0127] The correction module 34 is configured to correct the first hidden layer sequence according to the feature vector to generate a second hidden layer sequence corresponding to the description text.

[0128] The third generation module 35 is configured to input the second hidden layer sequence into a preset decoder to generate a target abstract corresponding to the description text.

[0129] In actual use, the abstract generation device provided by the embodiment of the present application can be configured in any electronic device to execute the abstract generation method described above.

[0130] The abstract generation device provided in the embodiments of the present application comprises: when an abstract generation request is acquired, inputting a description text in the abstract generation request into a preset first encoder to generate a first hidden layer sequence corresponding to the description text, and inputting an image into a preset second encoder to generate a feature vector corresponding to the image; then, correcting the first hidden layer sequence according to the feature vector to generate a second hidden layer sequence corresponding to the description text; and further inputting the second hidden layer sequence into a preset decoder to generate a target abstract corresponding to the description text. In this way, the hidden layer sequence corresponding to the description text is corrected by using the feature vector corresponding to the image to mine key information in the description text information through image information, thereby improving the accuracy of abstract generation.

[0131] In a possible implementation form of the present application, the description text comprises a plurality of first words, and the first generation module 32 comprises:

[0132] A first generation unit is configured to input each first word into a preset first encoder to generate a first hidden layer vector corresponding to each first word.

[0133] A second generation unit is configured to fuse the first hidden layer vectors corresponding to the plurality of first words to generate a first hidden layer sequence.

[0134] Further, in another possible implementation form of the present application, the correction module 34 comprises:

[0135] A correction unit is configured to correct each first hidden layer vector according to the feature vector to generate a second hidden layer vector corresponding to each first word.

[0136] A third generation unit is configured to fuse the second hidden layer vectors corresponding to the plurality of first words to generate a second hidden layer sequence.

[0137] Further, in another possible implementation form of the present application, the correction unit is specifically configured to:

[0138] determine a mean value of the plurality of first hidden layer vectors corresponding to the plurality of first words;

[0139] acquire a first weight matrix corresponding to the first hidden layer vector, a second weight matrix corresponding to the mean value of the first hidden layer vector, and a third weight matrix corresponding to the feature vector;

[0140] correct each first hidden layer vector according to the first weight matrix, the second weight matrix, the third weight matrix, the mean value of the first hidden layer vector, and the feature vector to generate a second hidden layer vector corresponding to each first word.

[0141] Further, in a possible implementation form of the application, the target abstract comprises a plurality of target abstract words, and the third generation module 35 comprises:

[0142] The first determination unit is configured to determine the attention score of each first segmented word at the t-th moment;

[0143] The second determination unit is configured to determine the context vector of the text at the t-th moment according to the attention score of each first segmented word at the t-th moment and each second hidden layer vector;

[0144] The third determination unit is configured to generate the t-th target hidden layer sequence according to the (t-1)-th target hidden layer sequence generated by the preset decoder at the (t-1)-th moment, the (t-1)-th target abstract word, and the context vector of the text at the t-th moment;

[0145] The acquisition unit is configured to acquire the t-th target abstract word from the preset dictionary according to the t-th target hidden layer sequence and the context vector of the text at the t-th moment;

[0146] The fourth generation unit is configured to generate the target abstract according to the target abstract word generated by the preset encoder at each moment.

[0147] Further, in a possible implementation form of the application, the first determination unit is specifically configured to:

[0148] determine the attention score of each first segmented word at the t-th moment according to the second hidden layer vector corresponding to each first segmented word and the (t-1)-th target hidden layer sequence.

[0149] Further, in a possible implementation form of the application, the acquisition unit is specifically configured to:

[0150] acquire the first parameter matrix and the second parameter matrix corresponding to each second segmented word in the preset dictionary;

[0151] determine the probability that each second segmented word is the t-th target abstract word according to the first parameter matrix and the second parameter matrix corresponding to each second segmented word, the t-th target hidden layer sequence, and the context vector of the text at the t-th moment;

[0152] determine the second segmented word with the maximum probability as the t-th target abstract word.

[0153] It should be noted that the above explanation of the abstract generation method embodiment shown in Figure 1 、 Figure 2 is also applicable to the abstract generation device 30 of the embodiment, and thus will not be repeated here.

[0154] The abstract generation device provided in the embodiments of the present application corrects the first hidden layer sequence corresponding to the description text according to the feature vector corresponding to the image, generates a second hidden layer sequence corresponding to the description text, determines the attention score of each first word segmentation at time t and the context vector of the description text at time t through the attention mechanism, then generates the tth target hidden layer sequence according to the t-1th target hidden layer sequence generated by the preset decoder at time t-1, the t-1th target abstract word, and the context vector of the description text at time t, further acquires the tth target abstract word from the preset dictionary according to the tth target hidden layer sequence and the context vector of the description text at time t, and generates the target abstract according to the target abstract word generated by the preset encoder at each time. In this way, the hidden layer sequence corresponding to the description text is corrected by using the feature vector corresponding to the image, so as to mine the key information in the description text information through the image information, and the attention mechanism is introduced in the decoding process, the word segmentation matched with the decoding result is selected from the preset dictionary to generate the target abstract, thereby further improving the accuracy of abstract generation.

[0155] To achieve the above-mentioned embodiments, the present application further provides an electronic device.

[0156] Figure 7 The structure diagram of the electronic device of one embodiment of the present application.

[0157] As Figure 7 shown, the above-mentioned electronic device 200 includes:

[0158] The memory 210 and the processor 220, the bus 230 connecting different components (including the memory 210 and the processor 220), the memory 210 stores a computer program, when the processor 220 executes the program, the abstract generation method described in the embodiments of the present application is realized.

[0159] The bus 230 represents one or more of several bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of the bus structures. For example, these architectures include but are not limited to industry standard architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video electronics standards association (VESA) local bus and peripheral component interconnect (PCI) bus.

[0160] The electronic device 200 typically includes various electronic device readable media. These media can be any available media that can be accessed by the electronic device 200, including volatile and non-volatile media, removable and non-removable media.

[0161] Memory 210 can also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. Electronic device 200 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 260 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Figure 7 Although not shown, a magnetic disk drive can also be utilized for reading from and writing to a removable, non-volatile magnetic media (such as a "floppy disk"), and an optical disk drive can be utilized for reading from and / or writing to a removable, non-volatile optical media (such as a CD-ROM, DVD-ROM or other optical media). Figure 7 In such instances, each drive can be connected to the bus 230 by one or more data media interfaces. The memory 210 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

[0162] Program / utility 280, having a set (at least one) of program modules 270, can be stored in memory 210 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data, each or some combination thereof, can include implementation of a networking environment. Program modules 270 generally carry out the functions and / or methodologies of embodiments of the application as described herein.

[0163] Electronic device 200 can also communicate with one or more external devices 290 such as a keyboard or pointing device, a display 291, etc.; one or more devices that enable a user to interact with electronic device 200; and / or one or more devices that enable electronic device 200 to communicate with one or more other computing devices. Such communication can be via input / output (I / O) interfaces 292. Similarly, such communication can be via network adapter 293. Network adapter 293 can communicate with the other

[0164] Processor 220 can execute instructions for various functions and data processing by running programs stored in memory 210.

[0165] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are described in the foregoing summary generation method of the embodiments of the present application, which will not be described here.

[0166] The electronic device provided by the embodiments of the present application can execute the summary generation method described above. When the summary generation request is obtained, the description text in the summary generation request is input into the preset first encoder to generate the first hidden layer sequence corresponding to the description text, and the image is input into the preset second encoder to generate the feature vector corresponding to the image. Then, the first hidden layer sequence is corrected according to the feature vector to generate the second hidden layer sequence corresponding to the description text, and then the second hidden layer sequence is input into the preset decoder to generate the target summary corresponding to the description text. Thus, the hidden layer sequence corresponding to the description text is corrected by using the feature vector corresponding to the image to mine the key information in the description text information through the image information, thereby improving the accuracy of the summary generation.

[0167] In order to achieve the above-mentioned embodiments, the present application further provides a computer readable storage medium.

[0168] The computer readable storage medium has a computer program stored thereon, and the program is executed by a processor to implement the summary generation method described in the embodiments of the present application.

[0169] In order to achieve the above-mentioned embodiments, the present application further provides a computer readable storage medium.

[0170] In an optional implementation form, the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus.

[0171] A computer readable signal medium can include a propagated data signal with computer executable code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport programming code.

[0172] Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0173] Computer program code for carrying out operations for aspects of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's electronic device, partly on the user's electronic device, as a stand-alone software package, partly on the user's electronic device and partly on a remote electronic device or entirely on the remote electronic device or server. In the latter scenario, the remote electronic device can be connected to the user's electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external electronic device (for example, through the Internet using an Internet Service Provider). The application is not limited to the above-mentioned embodiments and can be implemented in various ways.

[0174] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application be limited only by the scope of the claims, a properly construed, and it is intended that each definition in the claims be understood to embrace any and all equivalents. The specification and examples are illustrative only, and not intended to be limiting. Other objects, advantages and novel features of the application will become apparent from the following detailed description when made in connection with the

[0175] It is to be understood that the application is not limited to the precise details of construction and the arrangement of components described above and illustrated in the drawings and that various modifications and changes can be made without departing from the scope of the application. The scope of the application is limited only by the claims appended hereto.

Claims

1. A method for generating a summary, characterized in that, include: Obtain a summary generation request, wherein the summary generation request includes descriptive text and an image, and the descriptive text includes multiple first-segment words; The description text is input into a preset first encoder to generate the first hidden layer sequence corresponding to the description text; The image is input into a preset second encoder to generate a feature vector corresponding to the image; The first hidden layer sequence is modified based on the feature vector to generate the second hidden layer sequence corresponding to the descriptive text; The second hidden layer sequence is input into a preset decoder to generate a target summary corresponding to the description text; The step of modifying the first hidden layer sequence based on the feature vector to generate the second hidden layer sequence corresponding to the descriptive text includes: Each of the first word segments is input into a preset first encoder to generate a first hidden layer vector corresponding to each of the first word segments. Determine the mean of multiple first hidden layer vectors corresponding to multiple first word segments; Obtain the first weight matrix corresponding to the first hidden layer vector, the second weight matrix corresponding to the mean of the first hidden layer vector, and the third weight matrix corresponding to the feature vector; Based on the first weight matrix, the second weight matrix, the third weight matrix, the mean of the first hidden layer vectors, and the feature vector, each first hidden layer vector is modified to generate a second hidden layer vector corresponding to each first word segmentation. The second hidden layer vectors corresponding to multiple first word segments are fused to generate the second hidden layer sequence; The second hidden layer vector corresponding to the first word segment is determined by the following formula: in, This describes the second hidden layer vector corresponding to the i-th first word in the text. This describes the first hidden layer vector corresponding to the i-th first word in the text. Here, is the sigmoid function, N is the number of first-order words in the text, and i is the index of the first-order word in the text. v The feature vector corresponding to the image. This is the first weight matrix. This is the second weight matrix. This is the third weight matrix.

2. The method as described in claim 1, characterized in that, The target summary includes multiple target summary terms. The step of inputting the second hidden layer sequence into a preset decoder to generate the target summary corresponding to the description text includes: Determine the attention score for each of the first word segments at time t; The context vector of the descriptive text at time t is determined based on the attention score of each of the first word segments at time t and each of the second hidden layer vectors; Based on the (t-1)th target hidden layer sequence generated by the preset decoder at time t-1, the (t-1)th target summary word, and the context vector of the description text at time t, the tth target hidden layer sequence is generated. Based on the t-th target hidden layer sequence and the context vector of the description text at time t, the t-th target summary word is obtained from the preset dictionary; The target summary is generated based on the target summary words generated by the preset encoder at each time step.

3. The method as described in claim 2, characterized in that, Determining the attention score of each of the first word segments at time t includes: Based on the second hidden layer vector corresponding to each of the first segmented words and the (t-1)th target hidden layer sequence, determine the attention score of each of the first segmented words at time t.

4. The method as described in claim 2, characterized in that, The step of obtaining the t-th target summary word from a preset dictionary based on the t-th target hidden layer sequence and the context vector of the description text at time t includes: Obtain the first parameter matrix and the second parameter matrix corresponding to each second word in the preset dictionary; Based on the first parameter matrix and the second parameter matrix corresponding to each second word, the t-th target hidden layer sequence, and the context vector of the description text at time t, determine the probability that each second word is the t-th target summary word; The second word segment with the highest probability of being the t-th target summary word is determined as the t-th target summary word.

5. An apparatus for generating a summary, characterized in that, include: The acquisition module is used to acquire a summary generation request, wherein the summary generation request includes descriptive text and an image, and the descriptive text includes multiple first words; The first generation module is used to input the description text into a preset first encoder to generate a first hidden layer sequence corresponding to the description text. The second generation module is used to input the image into a preset second encoder to generate a feature vector corresponding to the image; The correction module is used to correct the first hidden layer sequence according to the feature vector to generate the second hidden layer sequence corresponding to the descriptive text; The third generation module is used to input the second hidden layer sequence into a preset decoder to generate the target summary corresponding to the description text; The step of modifying the first hidden layer sequence based on the feature vector to generate the second hidden layer sequence corresponding to the descriptive text includes: Each of the first word segments is input into a preset first encoder to generate a first hidden layer vector corresponding to each of the first word segments. Determine the mean of multiple first hidden layer vectors corresponding to multiple first word segments; Obtain the first weight matrix corresponding to the first hidden layer vector, the second weight matrix corresponding to the mean of the first hidden layer vector, and the third weight matrix corresponding to the feature vector; Based on the first weight matrix, the second weight matrix, the third weight matrix, the mean of the first hidden layer vectors, and the feature vector, each first hidden layer vector is modified to generate a second hidden layer vector corresponding to each first word segmentation. The second hidden layer vectors corresponding to multiple first word segments are fused to generate the second hidden layer sequence; The second hidden layer vector corresponding to the first word segment is determined by the following formula: in, This describes the second hidden layer vector corresponding to the i-th first word in the text. This describes the first hidden layer vector corresponding to the i-th first word in the text. Here, is the sigmoid function, N is the number of first-order words in the text, and i is the index of the first-order word in the text. v The feature vector corresponding to the image. This is the first weight matrix. This is the second weight matrix. This is the third weight matrix.

6. An electronic device, characterized in that, include: A memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, when the processor executes the program, it implements the digest generation method as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the digest generation method as described in any one of claims 1-4.

Images