A region-aware image-guided story continuation method

Abstract

This invention discloses a region-aware image-guided story continuation method. It extracts keywords from the story context to obtain story development clues, utilizes existing large-scale knowledge graph networks to extract story clues, and filters the knowledge graph by calculating its similarity to the input image. A scene graph is constructed from the input image. The filtered knowledge graph is matched with the image scene graph, and the features of matched image objects and their connected objects in the scene graph are extracted as key image features. The corresponding image regions are considered key image regions that conform to the story context logic. An image sentiment extractor trained on an image sentiment dataset is used to extract sentiment features from the key image regions. Using the story context, the filtered knowledge graph, the selected key image features, and the sentiment features of the key image regions, a more specific, consistent, and emotionally resonant story ending is generated.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and computer vision, specifically to a region-aware image-guided story continuation method. Background Technology

[0002] Image-guided story ending continuation tasks require generating a concluding sentence that conforms to both the contextual logic and the information conveyed by the given image, given an incomplete story context. The main challenge is to effectively utilize image and text information to generate a logically sound ending that satisfies the constraints of the image's content and emotional message. Early work primarily focused on pure text-based story ending continuation tasks. Due to the inherent constraints of this task, the generated endings were often generalized and lacked specificity. Later work proposed image-guided story ending continuation tasks, using a relevant image to constrain the generation of the ending content, resulting in more specific and meaningful outcomes. However, existing models only utilize the features of the entire image without considering the relationship between regions within the image and the story context, lacking specificity. Using an image as a guide for ending generation naturally requires ensuring that the generated ending's emotional message matches the emotional information conveyed by the image. Furthermore, common-sense knowledge is widely applied in text generation tasks to expand existing information and help the model deepen its understanding of the story's context. However, for image-guided story ending continuation tasks, the common sense knowledge obtained simply contains interfering information that does not match the image content. Therefore, it is necessary to filter this knowledge and select the knowledge that is relevant to the image content to help the model understand the input information of different modalities. Summary of the Invention

[0003] To address the shortcomings of existing methods for writing continuations of image-based stories—namely, the lack of emphasis on the relationship between image content and the story's logical context, the failure to consider the guiding role of image emotional features in the ending, and the neglect of the role of common sense—this invention proposes a region-aware image-guided method for writing continuations of stories. The specific scheme is as follows:

[0004] A region-aware image-guided story continuation method includes the following steps:

[0005] S1, input the story context X and the image I that leads to the ending respectively, extract keywords from the sentences in the story context, and connect the obtained keywords in order to form the story context development clue K;

[0006] S2, Extracting knowledge of keywords from the story context development clues K in step S1 from the large-scale knowledge base ConceptNet. The knowledge graph G is composed of R Computational knowledge graph G RThe similarity score between the concepts in the image and the input image I is used to select concepts whose similarity scores meet a threshold, thus obtaining the filtered knowledge graph.

[0007] S3, identify objects in the image and the relationships between them, and construct the scene graph G of the image. I ;

[0008] S4, the knowledge graph filtered in step S2 and the scene graph G obtained in step S3 I Perform matching to obtain the matched image objects.

[0009] S5, the scene graph G constructed from step S3 I In the middle, select the image object matched in step S4. and the objects directly connected to it All these object features serve as key features of the image.

[0010] S6, based on the image object in step S5 and the objects directly connected to it At the location (x, y, w, h) in the original image, calculate the key features corresponding to the selected image. Corresponding key image region I sub center coordinates and the key region I of the image sub width and high Obtain the coordinates of key regions in an image

[0011] S7, use an image sentiment extractor trained on an image sentiment dataset to obtain the key image regions I from step S6. sub emotional characteristics f senti ;

[0012] S8, utilize the story context X from step S1 and the filtered knowledge graph from step S3. Key image features selected in step S5 And the sentiment features f of key regions in the image obtained in S7 senti Generate the story ending.

[0013] Furthermore, the keyword identification tool RAKE is used to obtain the keywords in the context sentences of the story in step 1.

[0014] Furthermore, the common sense knowledge described in step S2 The expression is:

[0015]

[0016] Where H represents the entity used for querying, T represents the word related to H in ConceptNet, and R represents the relationship between the two;

[0017] The knowledge graph G R The expression is:

[0018]

[0019] Furthermore, the scene graph G of the image described in step S3 I The expression is:

[0020] G I =(N I V I )

[0021] Where, N I =(N I1 N I2 ,…,N Io V represents a group of objects identified in the image. I =(V I1 V I2 ,…,V Id () represents the relationship between a group of objects.

[0022] Further, the image object described in step S5 Directly connected objects The edge v formed between I The expression is:

[0023]

[0024] In the formula, r I (sbj,obj) This represents the image objects matched in step S5. and objects directly connected to it in the scene diagram The relationships that exist between them allow us to extract content information from key areas in the image that align with the story's development. I =(v I1 ,v I2 ,…,v It In the formula, t represents the scene graph G. I There are t edges to choose from.

[0025] Furthermore, the coordinates of the i-th image object in step S5 are represented as (x... i ,y i ,w i ,h i ). Where (x i ,yi ) represents the center coordinates of the object in the image, w i h is the width of the image object. i The height of the object in the image. Key features of the image. The corresponding key region I of the image sub center coordinates The calculation formula is:

[0026]

[0027]

[0028] Key region I of the image sub width and high The calculation formula is as follows:

[0029]

[0030]

[0031] Based on the calculated key region I of the image sub coordinates Obtain key regions of the image I sub .

[0032] Furthermore, the emotional features f of the key regions of the image in step S7 senti The method of obtaining it is:

[0033] f senti =DNN(I sub )

[0034] DNN is an image sentiment extractor trained on an image sentiment dataset.

[0035] Furthermore, step S8 utilizes the story context X from step S1 and the filtered knowledge graph from step S3. Key image features selected in step S6 The emotional features f of key regions in the image obtained in S7 senti The process of generating the story ending P(Y) is represented as follows:

[0036]

[0037] In the formula, y <t =y1y2...y t-1 , m indicates that the generated ending contains m words.

[0038] Advantages of the present invention

[0039] This invention provides a region-aware image-guided story continuation method, which is an LSTM-based Encoder-Decoder neural network algorithm. It effectively expands the common-sense knowledge associated with story clues using a knowledge graph, and filters the knowledge graph based on image content to obtain knowledge that is relevant to both the story context and the image content. This knowledge is beneficial for the model's understanding of input information from different modalities. Simultaneously, a scene graph is used to construct relationships between image objects. By matching the filtered knowledge graph with the image's scene graph, matched image objects are selected. The features of these objects and their directly connected image objects in the scene graph are key image features, and the regions formed by these objects are key image regions. In this way, image information can be used more effectively to guide the continuation of the story ending. Furthermore, this method uses an image sentiment extractor trained on an image sentiment dataset to obtain the sentiment features of key image regions, making the generated story ending more emotionally rich and more consistent with the high-level semantic information of the image. Since different regions in an image may convey different emotions, and the overall emotion of an image is not entirely consistent with local emotions, targeted extraction of the emotions of key image regions to guide the continuation of the story ending is crucial. By selectively acquiring external knowledge, image content features, and image emotional features, we can effectively expand existing information and make targeted use of image features to generate endings that are rich in content, logically sound, and emotionally consistent. Attached Figure Description

[0040] Figure 1 This is a flowchart of the region-aware image-guided story continuation method of the present invention.

[0041] Figure 2 Given a story context and an image related to the ending, this is a comparison chart of story continuations using different models. Detailed Implementation

[0042] The present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. However, it should be noted that these specific embodiments are not intended to limit the scope of the present invention.

[0043] like Figure 1 As shown in the example, this method for region-aware image-guided story continuation includes the following steps:

[0044] S1, input the story context X and an image I used to guide the ending, respectively. Use the RAKE keyword recognition tool to extract keywords from the sentences in the story context to highlight key information in the story context; the story context keywords refer to entities and events in the illustrated text, words that can convey specific information.

[0045] The keywords obtained are connected in sequence to form the development clue K of the story context. The development clue K of the story context is used to simplify the story content, highlight the development trend of the story context, and provide a basis for the subsequent purposeful selection of image content and emotional information.

[0046] The order of appearance refers to the sequence in which the obtained entities appear in the story context.

[0047] The story context development clue K is composed of these acquired keywords in chronological order. It includes the beginning, development, climax, and turning point of the story. The model will generate a reasonable ending that conforms to the development trend of the story context development clue K.

[0048] A reasonable ending must first and foremost conform to the story's development trend, while also adhering to the constraints of the image's content. Therefore, constructing the story's contextual clues (K) can provide a basis for subsequently selecting key areas of information from the image.

[0049] S2. Considering that in reality, the keywords in the story's contextual clues K may not be directly identical to the objects in the image, but people can understand the connection between them through existing knowledge and experience, we utilize the existing large-scale knowledge graph network ConceptNet to extract knowledge from the keywords in the story clues. Common sense knowledge The expression is:

[0050]

[0051] In the formula, H represents the entity used for querying, T represents the word related to H in ConceptNet, and R represents the relationship between the two. H and T are actual English words, such as dog and human.

[0052] Knowledge extracted from keywords in the development clues of the story context K The knowledge graph G is composed of R The expression is:

[0053]

[0054] However, directly obtained knowledge graphs contain many concepts unrelated to the image content, which may introduce noise. Therefore, a pre-trained CLIP model is used to calculate the similarity score between concepts in the knowledge graph and the input image. Concepts with similarity scores meeting a threshold are selected to obtain a filtered knowledge graph. Filtered knowledge graph The concepts in the text are based on the context of the story and have a certain similarity to the image content, which can effectively establish the relationship between different types of input information.

[0055] S3 identifies objects in the image and the relationships between them, and constructs the scene graph G of the image based on this information. I G I The expression is:

[0056] G I =(N I V I )

[0057] In the formula, N I =(N I1 N I2 ,…,N Io V represents a group of objects identified in the image. I =(V I1 V I2 ,…,V Id () represents the relationship between a group of objects.

[0058] This method uses a trained image recognition model to capture objects in an image, returning the object's features, label, and location information. Then, it extracts relationship features between the captured image objects and selects the relationship category with the highest probability as the object relationship information.

[0059] S4, the knowledge graph filtered in step S2 and the scene graph G obtained in step S3 I Perform a match to obtain the matching image objects that are related to the story context clue K.

[0060] The matching process involves determining whether the label of an image object exists in the knowledge graph of keywords that constitute the story's contextual development clue K. If it exists, the object is considered to conform to the development trend of the story's contextual development clue K; otherwise, the object is considered not to conform to the development trend of the story's contextual development clue K.

[0061] S5, the scene graph G constructed from step S3 I In the process, select the image object that matches the one found in step S4. Directly connected objects Features as key image features The edge v formed between them I Represented as:

[0062]

[0063] In the formula, r I (sbj,obj)This represents the image objects matched in step S5. and objects directly connected to it in the scene diagram The relationships that exist between them allow for the targeted acquisition of content information from key areas of the image that align with the story's development. I =(v I1 ,v I2 ,…,v It ), where t represents the number of edges selected from the scene graph.

[0064] S6, Obtain the image object from step S5. Directly connected objects At the location (x, y, w, h) in the original image, calculate the key features corresponding to the selected image. Corresponding key image region I sub center coordinates

[0065]

[0066]

[0067] Key region I of the image sub width and high The calculation formula is as follows:

[0068]

[0069]

[0070] In the formula, the coordinates of the i-th image object are represented as (x... i ,y i ,w i ,h i ), where (x i ,y i ) represents the center coordinates of the object in the image, w i h is the width of the image object. i The height of the object in the image.

[0071] Acquired image key region I sub The coordinates are The key region I of the image sub Image key areas I that conform to the logical development of the story context sub .

[0072] S7, Emotional features of key regions in an image f senti The method of obtaining it is expressed as follows:

[0073] f senti=DNN(I sub )

[0074] DNN is an image sentiment extractor trained on an image sentiment dataset.

[0075] The purpose of capturing image emotion is to: 1) enrich the emotional expression of the generated ending, making it more attractive; and 2) utilize the constraints of image emotion to make the generated ending more consistent with the image content, avoiding the generation of endings that clearly do not conform to the information conveyed by the image.

[0076] S8, utilize the story context X from step S1 and the filtered knowledge graph from step S3. Key image features selected in step S6 The emotional features f of key regions in the image obtained in S7 senti The process of generating the story ending P(Y) is represented as follows:

[0077]

[0078] Among them, y <t =y1y2...y t-1 , m indicates that the generated ending contains m words.

[0079] The following examples illustrate how to continue a story using the methods described above. Figure 2 As shown:

[0080] Figure 2 The upper part of the diagram shows the given story context, with bolded words representing keywords. The image in the upper right corner is a related image for the given ending. The upper knowledge graph displays the knowledge graph of keywords within the story context. The lower part shows the results generated by different models in the story ending continuation task and the image-guided story ending continuation task. Bolded words are words that have already appeared in the story context, words under a single solid line are words predicted by the model but are not entirely reasonable, and words under double solid lines are words predicted by the model that are reasonable.

[0081] according to Figure 2Given a story context and an image related to the ending, the model in this embodiment can obtain a knowledge graph from keywords in the story context, filter the knowledge graph based on the given image, and then selectively select key regions from the image to guide the generation of the ending. The generated results show that the model in this embodiment can accurately identify the groom, bride, emotional state of happiness, and the related event of marriage. Compared to the results generated by other models, our model's results are more fluent and reasonable, with a higher degree of overlap with the given ground truth. Compared to story ending continuation, image-guided story ending continuation generates richer, more specific, and more vivid ending information.

[0082] This invention is compared with existing technologies:

[0083] The following section evaluates a region-aware image-guided story continuation method from this embodiment on the VIST-E corpus. The VIST-E public dataset contains 49,313 stories, each consisting of a four-sentence story context, an image related to the ending, and a story conclusion. The dataset is divided into training, validation, and test sets, with 39,320, 4,963, and 5,030 stories respectively.

[0084] In the evaluation phase, both automatic and manual evaluation metrics were used to compare and evaluate the method of this embodiment with existing technologies. In the automatic evaluation phase, BLEU-n (Bn), CIDEr (C), METER (M), and ROUGE-L (RL) evaluation metrics were used to measure the consistency between the generated sentence and the reference sentence. In the manual evaluation phase, Grammer was used to evaluate the fluency and coherence of the generated sentence; Logicality was used to evaluate the logical consistency between the generated sentence and the story context; Relevance was used to evaluate the consistency between the generated sentence and the image content; and Sentiment-Coherence (SC) was used to evaluate the consistency between the generated sentence and the image sentiment. Higher values ​​for all these evaluation metrics are better.

[0085] During training on the VIST-E corpus, a learning rate of 3e-5 and the Adam optimizer were used for 20 epochs. The LSTM decoder used in the generation process had a hidden layer dimension of 512. The training objective was to minimize the cross-entropy loss function. Compared with existing techniques, this embodiment achieved state-of-the-art results on almost all evaluation metrics.

[0086] Table 1 shows a comparison of the evaluation metrics between the method of this embodiment and other prior art methods:

[0087] Table 1:

[0088] Model B1 B4 C M RL Gram. Logic. Rele. SC. Seq2Seq 16.06 1.72 12.21 4.62 16.92 0.82 0.57 0.48 0.71 Transformer 17.47 1.55 12.87 7.03 18.41 0.87 0.59 0.54 0.73 IE+MSA 19.64 1.73 15.38 6.47 20.77 0.96 0.71 0.57 0.84 T-CVAE 19.57 2.07 14.53 6.58 17.09 0.92 0.65 0.63 0.81 MGCN-DP 21.47 1.92 19.81 7.48 20.03 1.02 0.75 0.67 0.87 MGCL 22.57 2.49 21.46 7.84 21.66 1.06 0.74 0.85 0.97 This embodiment 23.15 2.30 22.39 8.07 22.32 1.10 0.80 0.91 1.12

[0089] The results of this embodiment show that, in the automatic evaluation metrics, our model achieved the best results on B1, C, M, and RL, and was second only to the latest MGCL model on the B4 metric. This indicates that the results generated by our model have a higher degree of overlap with the real results. In the manual evaluation metrics, our model improved in terms of grammatical correctness, logical consistency, relevance, and sentiment consistency, producing more reasonable story endings that are more consistent with the content of real story endings.

[0090] The above embodiments are merely preferred examples of the present invention and are not intended to limit the invention. Those skilled in the art can make various modifications and variations based on the present invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are within the scope of protection of the present invention.

Claims

1. A region-aware image-guided story continuation method, characterized in that, Includes the following steps: S1, Input the story context respectively and the image that leads to the end Extract keywords from sentences in the story's context, and connect these keywords in chronological order to form a narrative thread that guides the story's development. ; S2, Extracting the story context development clues from the large-scale knowledge base ConceptNet in step S1. Chinese keyword knowledge The knowledge graph is composed of Computational knowledge graph The concept and the input image The similarity scores between the data points are used to select those that meet a threshold, and the resulting filtered knowledge graph is obtained. ; S3 identifies objects in an image and the relationships between them, constructing a scene graph of the image. ; S4, the knowledge graph filtered in step S2 and the scene map obtained in step S3 Perform matching to obtain the matched image objects. ; S5, the scene graph constructed from step S3 In the middle, select the image object matched in step S4. and the objects directly connected to it Features as key features of an image ; S6, Obtain the image object obtained in step S5. and the objects directly connected to it Location in the original image Calculate key features of the image Corresponding key regions of the image center coordinates and the key areas of the image width and high Obtain key regions of the image coordinates ; S7, use an image sentiment extractor trained on an image sentiment dataset to obtain the key regions of the image from step S6. Emotional characteristics ; S8, utilizing the story context of step S1 The knowledge graph after filtering in step S3 Key image features selected in step S5 And the emotional features of key image regions acquired in S7. Generate the story ending; The use of the story context from step S1 in step S8 is described in step S8. The knowledge graph after filtering in step S3 Key image features selected in step S6 Sentiment features of key regions in images obtained from S7 Generate story ending The process is represented as: ， In the formula, , This indicates that the generated end contains One word.

2. The region-aware image-guided story continuation method according to claim 1, characterized in that, Step 1: Obtain keywords from the context sentences of the story using the RAKE keyword recognition tool.

3. The region-aware image-guided story continuation method according to claim 1, characterized in that, Keyword knowledge described in step S2 The expression is: ， in, Indicates the entity used for querying. Indicating that in ConceptNet and Related words, This indicates the relationship between the two. The knowledge graph The expression is: 。 4. The region-aware image-guided story continuation method according to claim 1, characterized in that, The scene diagram of the image described in step S3 The expression is: ， in, This represents a group of objects identified in an image. It represents the relationship between a group of objects.

5. The region-aware image-guided story continuation method according to claim 1, characterized in that, The image object described in step S5 Directly connected objects The edges formed between The expression is: ， In the formula, This indicates the image objects matched in step S5. and objects directly connected to it in the scene diagram By understanding the existing relationships, we can extract content information from key areas of the image that align with the story's development. In the formula, Represents the scene diagram The selected edge has indivual.

6. The region-aware image-guided story continuation method according to claim 1, characterized in that, Key image features described in step S5 The center coordinates of the corresponding key region in the image The calculation formula is: ， ， Key areas of the image width and high The calculation formula is as follows: ， ， In the formula, Indicates the first The center coordinates of the objects in the image. For the first The width of the image object, For the first The height of the image object.

7. The region-aware image-guided story continuation method according to claim 1, characterized in that, Emotional features of key regions in the image in step S7 The method of obtaining it is: ， in This is an image sentiment extractor trained on an image sentiment dataset.

Images