Automatically generating storyboards

Abstract

The present disclosure provides methods and apparatus for automatically generating storyboards. A text including a plurality of sentences can be obtained. An image sequence corresponding to the plurality of sentences can be retrieved. The image sequence can be rendered to obtain a rendered image sequence. A storyboard including the rendered image sequence can be provided.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a method and apparatus for automatically generating a storyboard. BACKGROUND

[0002] A storyboard is a visual representation of a textual story. Generally, a storyboard can include a sequence of images that visually represent a textual story having a plurality of sentences, so that the story content can be vividly expressed image by image. A storyboard generation process is one of the most important stages for creating various story-related products, such as a movie, an animation, etc. A storyboard not only simplifies the understanding of a textual story with visual aids, but also makes subsequent creation steps of a story-related product more intuitive and smooth by planning key images in advance. SUMMARY

[0003] This summary is provided to introduce a set of concepts that are further described below in the of the Invention. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in limiting the scope of the claimed subject matter.

[0004] Embodiments of the present disclosure propose a method and apparatus for automatically generating a storyboard. A text including a plurality of sentences can be obtained. A sequence of images corresponding to the plurality of sentences can be retrieved. The sequence of images can be rendered to obtain a rendered sequence of images. A storyboard including the rendered sequence of images can be provided.

[0005] It should be noted that the above one or more aspects include the features recited in the following detailed description and the claims. The following description and drawings detail certain illustrative features of the one or more aspects. These features are indicative of but a few of the various ways in which the principles of various aspects can be employed. This disclosure is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF DRAWINGS

[0006] Various aspects disclosed herein will be described in connection with the following figures, in which:

[0007] Figure 1 An exemplary architecture for automatically generating a storyboard according to an embodiment is shown.

[0008] Figure 2 An example of automatically generating a storyboard according to an embodiment is shown.

[0009] Figure 3 An exemplary structure of a context-aware encoder according to an embodiment is shown.

[0010] Figure 4An exemplary image retrieval augmentation process according to an embodiment is shown.

[0011] Figure 5 An exemplary related image region preservation process according to an embodiment is shown.

[0012] Figure 6 An example of performing a related image region preservation process according to an embodiment is shown.

[0013] Figure 7 An exemplary character image element replacement process according to an embodiment is shown.

[0014] Figure 8 An exemplary scene image element replacement process according to an embodiment is shown.

[0015] Figure 9 An example of retrieving and rendering a sequence of images according to an embodiment is shown.

[0016] Figure 10A and Figure 10B An example of retrieving and rendering a sequence of images according to an embodiment is shown.

[0017] Figure 11 A flowchart of an exemplary method for automatically generating a storyboard according to an embodiment is shown.

[0018] Figure 12 An exemplary apparatus for automatically generating a storyboard according to an embodiment is shown.

[0019] Figure 13 An exemplary apparatus for automatically generating a storyboard according to an embodiment is shown. DETAILED DESCRIPTION

[0020] The present disclosure will now be discussed with reference to a number of exemplary embodiments. It should be understood that the discussion of these embodiments is merely meant to provide a better understanding of the present disclosure and to enable the best possible implementation of the embodiments of the present disclosure, and is not meant to teach any limitation of the scope of the present disclosure.

[0021] The creation of storyboards presents a number of challenges. For example, the images in a storyboard should be cinematic, which takes into account framing, structure, perspective, etc. For example, the visualized images in a storyboard should contain sufficient details for telling the story, such as scenes, characters, actions of characters, etc. For example, the images in a storyboard should visually appear to have a consistent style and character image, etc.

[0022] Various methods for storyboarding have been proposed, including generative-based methods, retrieval-based methods, etc. Existing generative-based methods can directly generate images conditioned on text by, for example, generative adversarial networks (GANs), which can flexibly generate new images. However, due to known difficulties in training, generative-based methods struggle to generate high-quality, diverse, and relevant images, and are often limited to specific domains to generate simple-structured images, thus cannot be effectively applied to generate image sequences for a story in complex scenarios such as storyboarding. Existing retrieval-based methods can retrieve existing high-quality images based on text. However, most retrieval-based methods can only use a single sentence to retrieve images, without considering context. Since there is no guarantee that existing images are accurately relevant to the input story, retrieval-based methods lack flexibility. Existing retrieval-based methods use global or dense visual-semantic matching between sentences and images. Global visual-semantic matching uses fixed-dimensional global vectors for global visual or textual representations, but global vectors struggle to capture fine-grained information. Dense visual-semantic matching can represent images or sentences as a set of fine-grained components, but still only considers the matching of a single sentence and image. Furthermore, the retrieved images can come from different sources or databases, resulting in visual inconsistency in the image sequences in the storyboard in terms of style and character appearance.

[0023] Embodiments of the present disclosure propose to automatically generate storyboards based on a heuristic-creation architecture, e.g., to automatically generate a sequence of images to visually present a textual story including multiple sentences. Given that human understanding of language is a simulation of past experiences about vision, sound, etc., the heuristic-creation architecture according to embodiments of the present disclosure can take well-designed existing images as heuristics, and then render the retrieved images to create flexible visualized story content. For example, the heuristic-creation architecture can include a text-to-image retriever to retrieve or extract relevant images for heuristics. The heuristic-creation architecture can also include a storyboard creator to further optimize or render the retrieved images to improve relevance, visual consistency, etc. The text-to-image retriever can employ a context-aware dense visual semantic matching model that is capable of dynamically employing context information in a story with a hierarchical attention mechanism, and applying more interpretable dense visual semantic matching to accurately retrieve images and ground image regions. The text-to-image retriever can not only achieve accurate image retrieval, but also utilize multiple images to visually present one sentence. The storyboard creator can enhance flexibility of the retrieved images through a rendering process, which can include, for example, relevant image region preservation processing, style unification processing, element replacement processing, etc. The storyboard creator not only improves flexibility, but also improves relevance, visual consistency, etc. of the generated storyboards.

[0024] With embodiments of the present disclosure, better visualized presentation of textual stories can be achieved in open domains, and high-quality storyboards are provided.

[0025] Figure 1 An exemplary architecture 100 for automatically generating storyboards according to embodiments is shown. The architecture 100 can be an instance of the heuristic-creation architecture described above. The architecture 100 can be employed in an apparatus for automatically generating storyboards. In one implementation, the architecture 100 can include a text-to-image retriever 110, a storyboard creator 120, etc.

[0026] A text 102 to be used to generate a storyboard can be obtained first. The text 102 can include multiple sentences. The content of the text 102 can be a story, a movie script, etc.

[0027] The text-to-image retriever 110 can be used to retrieve a sequence of images corresponding to the multiple sentences in the text 102.

[0028] The text-to-image retriever 110 can include a context-aware encoding module 112. The context-aware encoding module 112 can generate a sequence of word representations for each sentence in the text 102 by performing context-aware encoding on the sentence. For example, the sequence of word representations corresponding to a sentence can be a set of word representations for all the words in the sentence. In this document, a "word" can refer to one or more characters, a single word, or a combination of multiple words, etc. The context-aware encoding module 112 can utilize relevant context in the text 102 to dynamically encode the sentence. The context-aware encoding can be based on a hierarchical attention mechanism. Through the hierarchical attention mechanism, a relevant cross-sentence context in the text 102 can be selected for each word in the sentence, and can in turn be weighted. In many cases, the cross-sentence context can be helpful for a complete and accurate understanding of the semantics of the sentence. For example, assume the current sentence is "they walked together in the forest," in which no specific explanation is made for the expression "they." If the previous sentence is "a little boy and his father went on a picnic," then it can be determined that "they" in the current sentence actually refers to "a little boy and his father" based on the context information provided by the previous sentence. As such, more relevant images for the current sentence can be retrieved in subsequent steps, such as images that present a little boy and a man.

[0029] After a sequence of word representations is generated for a sentence, the dense visual semantic matching module 114 in the text-to-image retriever 110 can select at least one image corresponding to the sentence based on the sequence of word representations. The images can be retrieved from a pre-created image library 130. The image library 130 can include images from various data sources, such as images in movies, images in cartoons, images scraped from the web, etc. The images in the image library 130 can be candidate images from which the dense visual semantic matching module 114 can retrieve at least one image corresponding to a sentence. In an implementation, the dense visual semantic matching module 114 can not only retrieve an image relevant to each sentence from the image library 130 through dense visual semantic matching, but also locate each word in the sentence to the most relevant image region in the retrieved image, which is also referred to as locating the image region. The determined locating image region can be further used in subsequent rendering processes. In an implementation, to cover as many details in the text as possible in the storyboard, when a sentence is long, the dense visual semantic matching module 114 can retrieve more than one image for a sentence, i.e., one-to-many image retrieval, so that multiple images can be utilized to visualize the sentence.

[0030] The text-to-image retriever 110 can retrieve one or more images for each sentence in the text 102, and all the retrieved images form the image sequence 104 corresponding to the text 102 or multiple sentences in the text 102.

[0031] The storyboard creator 120 can perform a rendering process on the image sequence 104 to obtain the rendered image sequence 106. The rendering process can enhance the flexibility of the retrieved images, and improve the relevance, visual consistency, etc. of the generated storyboard. The rendering process performed by the storyboard creator 120 can include at least one of a relevant image region preservation process 122, a style unification process 124, and an element replacement process 126.

[0032] The relevant image region preservation process 122 can preserve relevant image regions associated with words in a sentence in the images retrieved for the sentence, and / or remove irrelevant image regions not associated with any word in the sentence. Taking the sentence “a little boy walks in the forest” as an example, the images retrieved for the sentence can include image region “little boy”, image region “forest”, image region “puppy”, etc. Since the image region “little boy” and the image region “forest” are respectively associated with the words “little boy” and “forest” in the sentence, they can be preserved as relevant image regions, and since the image region “puppy” is not associated with any word in the sentence, it can be removed as an irrelevant image region.

[0033] The style unification process 124 can unify different styles of multiple images in the image sequence 104. In this context, the style of an image can refer to the form or effect of the image based on different criteria, such as color, contour, line, etc., or cartoon, sketch, real person picture, etc. Different images in the image sequence 104 can come from different sources, and thus can have different styles. In order to enhance the visual consistency of different images in the finally generated storyboard, the style unification process 124 can be performed on the image sequence 104 to enable multiple images in the image sequence 104 to have a consistent style. For example, if the image sequence 104 includes color images, black and white images, cartoon images, sketch images, etc., by performing the style unification process 124, these different style images in the image sequence 104 can be unified into a desired style, such as a cartoon style.

[0034] The element replacement processing 126 can replace or unify specific elements in the image sequence 104. In this context, an element in an image can be, for example, a character image element, a scene image element, etc., and can include one or more image regions in the image. A character image element can refer to one or more image regions in an image that exhibit a character, where a character can refer to a primary object in the image, such as a person, an animal, etc. A scene image element can refer to one or more image regions in an image that exhibit a scene. Thus, the element replacement processing can actually include character image element replacement processing, scene image element replacement processing, etc.

[0035] For the same character in the text 102, the retrieved images from different sources can have character image elements that take on different character appearances. For example, for a character "boy" in the text 102, different images in the image sequence 104 can include character image elements of the "boy" with different appearances from different movies. Through the element replacement processing 126, at least one character image element in the image sequence 104 that corresponds to the same character can be identified, and the identified at least one character image element can be replaced with a target character image element that is based on a specific target character appearance. The target character appearance can be a character appearance that is specifically designed for the character in the storyboard. The target character image element used to replace the character image element can have a pose, etc. that corresponds to the character image element, where a pose can refer to an action, an expression, a position, an orientation, etc.

[0036] The text 102 can include a description of a scene. Using the example of the sentence "a fierce wind is blowing and a little boy is walking in the forest with difficulty," the words "a fierce wind is blowing," "forest," and the like describe a scene. Accordingly, the images retrieved in response to the sentence can include a scene image element "forest," such as a picture of a number of large trees. Through the element replacement process 126, at least one scene image element associated with the scene in the image sequence 104 can be identified, at least one target scene image element associated with the scene can be selected from the pre-created image library, and the at least one scene image element associated with the scene in the image sequence 104 can be replaced with the target scene image element associated with the scene. Continuing with the example above, for instance, although the retrieved images include the scene image element "forest," such a scene image element can not fully represent the scene described by the combination of "a fierce wind is blowing" and "forest," such as "a forest in a fierce wind." However, through the element replacement process 126, a target scene image element representing "a forest in a fierce wind" can be retrieved from the pre-created image library, such as one depicting trees swaying in a strong wind, and the target scene image element can be used to replace the scene image element "forest" in the image sequence 104. The image after the scene image element replacement will be able to more fully represent the scene described in the sentence. It should be appreciated that the element replacement process 126 for scene image element replacement can also aim to unify scene image elements of different styles. For instance, various candidate target scene image elements in the pre-created image library can have a unified style, such that the scene image element replacement performed based on the pre-created image library will also result in the image sequence 104 being rendered into an image sequence of target scene image elements having a unified style.

[0037] The rendered image sequence 106 can further be used to form a storyboard 108. In one implementation, the rendered image sequence 106 can be properly arranged and directly serve as the storyboard 108. In one implementation, a textual description, such as the sentence in the text 102, can be added to the rendered image sequence 106 to form the storyboard 108. In one implementation, any other information that can be helpful to the creation of a story-related product can be added to the rendered image sequence 106 to form the storyboard 108.

[0038] It should be appreciated that, Figure 1The text-to-image retrieval unit 110 and storyboard author 120 in architecture 100 can be implemented in various ways, such as hardware, software, or a combination thereof. For example, in a hardware implementation, the text-to-image retrieval unit 110 and storyboard author 120 can be implemented using, for example, a dedicated processor or controller. Furthermore, in a software implementation, the text-to-image retrieval unit 110 and storyboard author 120 can be implemented as corresponding software modules or computer applications.

[0039] Figure 2 An example 200 of automatically generating storyboards according to an embodiment is shown.

[0040] Suppose we want to generate a storyboard for text 210. Text 210 may include, for example, statement 210-1, statement 210-2, ..., statement 210-N, etc.

[0041] It can be done through, for example Figure 1 The text-to-image retrieval unit 110 retrieves image sequences 220 corresponding to statements in the text 210. Image sequences 220 may include, for example, image 220-1 corresponding to statement 210-1, image 220-2 corresponding to statement 210-2, ..., image 220-N corresponding to statement 210-N, etc.

[0042] It can be done through, for example Figure 1 The storyboard creator 120 performs rendering processing on the image sequence 220 to obtain a rendered image sequence 230. The rendering processing may include at least one of relevant image region preservation processing, style unification processing, and element replacement processing. For example, by relevance image region preservation processing, relevant image regions related to "little boy," "suitcase," etc., in image 220-2 are preserved in image 230-2, while other irrelevant image regions are removed. For example, by style unification processing, the images in image sequence 230 have a unified watercolor painting style.

[0043] Storyboard 240 can be further formed based on the rendered image sequence 230. Storyboard 240 includes, exemplarily, the image sequence 230 and a text description for each image, wherein the text description may be derived from statements in text 210.

[0044] It should be understood that Figure 2 All statements, images, etc. in this document are exemplary and are only used to help intuitively understand the processing stages involved in the embodiments of this disclosure, and are not intended to limit the embodiments of this disclosure in any way.

[0045] Exemplary implementations of various processes for automatically generating storyboards according to embodiments of this disclosure will now be discussed in detail.

[0046] Assume S = {S 1 ,S 2 ,…,S N} represents a text story consisting of N statements, each statement being S. i It can be represented as a sequence of words, for example Where, n i For statement S i The number of words included. The purpose of storyboard generation is to generate an image sequence i = {I} for visualizing the text S. 1 ,I 2 ,…,I M The number of images, M, may or may not be equal to the number of statements, N.

[0047] Context-aware encoding

[0048] According to embodiments of this disclosure, context-aware encoding can be performed on the text in text-based image retrieval processing. Context-aware encoding can encode each word in a sentence by utilizing the necessary context within the sentence and the necessary context across sentences. Figure 3 An exemplary structure of a context-aware encoder 300 according to an embodiment is shown. Figure 3 The context-aware encoder 300 shown may include, for example, a word encoding layer, a sentence encoding layer, an enhanced sentence encoding layer, a joint encoding layer, etc. The context-aware encoder 300 may also include a diversity attention mechanism, which may include, for example, a first attention level and a second attention level. Figure 3 The structure and processing of the context-aware encoder 300 are illustrated using the encoding of the i-th statement in the text as an example.

[0049] Suppose the i-th statement is represented as S i And including n i words A word encoding layer can convert words into word embedding vectors in a vector space by performing word embeddings on each word. For example, a word encoding layer can perform word embeddings using a word embedding matrix We.

[0050] At the statement encoding layer, for words Bidirectional Long Short-Term Memory (LSTM) can be used to capture the information in a single statement S. i The context information within is as follows:

[0051] Where [·] represents vector concatenation, W h b h These are the parameters for learning. It is for words The word representations with contextual information within the sentence are obtained. Furthermore, the representation for the sentence can be obtained at the sentence encoding layer by averaging the word representations of the words within each sentence, as follows:

[0052]

[0053] Since the cross-sentence context of each word may differ, and the distribution of the cross-sentence context used to understand each word may also differ, the context-aware encoder 300 can employ a hierarchical attention mechanism to capture cross-sentence context. The first attention layer dynamically selects the relevant cross-sentence context for each word. For example, from sentence S in the text... 1 To statement S N The statement representations h obtained at the statement encoding layer respectively 1 ... h N In the context of statement S i Each word in the selection is related to the cross-sentence context. as follows:

[0054]

[0055] Where δ is a nonlinear ReLU function, and It is a parameter. Given a word representation from the statement encoding layer. and its cross-statement context The second attention layer adaptively weights the importance of each word across the sentence context, as follows:

[0056]

[0057] Where σ is the sigmoid function, and It is a parameter. Therefore, It is a word Word representations that have relevant cross-statement context information.

[0058] Optionally, to further distribute updated word representations within a single statement. In the enhanced sentence encoding layer, a bidirectional LSTM similar to that in Equation (1) can be used, which generates a context-informed word representation for each word.

[0059] In the joint coding layer, linear transformation can be used to... Transform into word representations in a joint visual semantic embedding space as follows:

[0060]

[0061] Among them, W x and b x These are the parameters used for linear mapping. In the manner described above, It will be encoded using both intra-statement context and cross-statement context.

[0062] The statement S is obtained through the context-aware encoder 300. i Word representation of each word in the text Then, the sequence of words represented by these words can be represented. Used for statement S i A fine-grained representation.

[0063] Dense visual semantic matching

[0064] After obtaining statement S i After obtaining the word representation sequence, an image corresponding to the sentence can be selected from candidate images in the image library based on the word representation sequence. An image I... j Represented as a set of fine-grained image region representations in a common visual semantic space Where, m j This refers to the number of image regions included in the image. Image regions can be detected, for example, using a bottom-up attention network, so that each image region can represent objects, object relationships, scenes, etc.

[0065] Based on S i and I j Dense representations and similarity of cross-modal pairs at each fine granularity Dense matching can be applied to calculate the global statement-image similarity F(S). i ,I j ),as follows:

[0066] Here, f(·) can be, for example, cosine similarity. Dense matching can locate each word to the most relevant image region in the image, and then the word-image region similarity can be averaged over all words to obtain the global sentence-image similarity.

[0067] Training dense visual semantic matching can employ a contrastive loss function, as follows:

[0068] L i =max(0,Δ-F(S) i ,I i )+F(S i ,I j ))+max(0,Δ-F(S i ,I i )+F(Sj I i )) Equation (7) where Δ is a pre-set hyper-parameter, (S i I i ) is a matching pair, while (S i I j ) and (S j I i ) are non-matching pairs. The total loss function is L i over all pairs in the training set.

[0069] The dense visual semantic matching described above can be used to select relevant images for each sentence, and can localize each word in the sentence to the most relevant image region in the image.

[0070] One-to-many image retrieval

[0071] In one implementation, to cover as many details in the text as possible, the dense visual semantic matching can select more than one image for a sentence, i.e., one-to-many image retrieval. Multiple images can be automatically selected by, for example, a greedy decoding algorithm, in order to enhance the coverage of the text content.

[0072] Figure 4 An exemplary image retrieval enhancement process 400 according to an embodiment is shown. The image retrieval enhancement process 400 can be performed for one-to-many image retrieval.

[0073] At 410, the sentences 402 in the text can be divided into multiple segments. In this context, a segment can refer to a combination of one or more consecutive words in a sentence that can express a relatively complete semantic. For example, the sentences 402 can be divided into multiple segments by constituency parsing.

[0074] At 420, a segment representation for each of the multiple segments can be obtained. In one implementation, the segment representation for a segment in a sentence can be obtained based on the word representation sequence of the sentence obtained by Figure 3 For example, a set of word representations of multiple words included in a segment can be used as the segment representation of the segment.

[0075] At 430, multiple images 404 or image sequences can be selected from the image library 440 corresponding to each segment by dense visual semantic matching. For each segment, a similarity between the segment and candidate images in the image library 440 can be computed, and the images can be selected based on the similarity. In one implementation, a greedy decoding algorithm can be used in the process of selecting images by dense visual semantic matching. For example, for each segment, the top k images can be first tried to be greedily selected, which are typically similar. If the k images have not been retrieved for previous segments, additional images from the k images can be utilized to visualize the segment to cover more details in the sentence. Otherwise, if the k images retrieved for the segment have been retrieved for previous segments, the images would be redundant and need not be used again.

[0076] Table 1 below shows an exemplary image retrieval enhancement process based on a greedy decoding algorithm.

[0077] The process in Table 1 aims to select an image sequence I c from a candidate image set I c ′.

[0078]

[0079]

[0080] Table 1

[0081] At step 1.1, the sentence S can be divided into n p segments. At step 1.2, a similarity between a segment p t and an image I j can be computed in a similar way as formula (6). At step 1.3, I c ′ and I r ′ are initialized, where I r ′ can be used in the following process to collect images that have been selected for previous segments. At step 1.4, a loop process starts for each of the n p segments. At step 1.5, if it is determined that F(p t ,I j ) is one of the top k similarities computed for the segment p t in the similarities between p t and each image, i.e., the image I j is one of the k images most relevant to the segment p t , the image I g is included in I j , where Ig The k images that can be used to collect the segment p t Most relevant images. It should be understood that the processing at step 1.5 can be performed on all n c candidate images in I c At step 1.6, it can be determined whether the k images collected in I g for the segment p t have already been retrieved for a previous segment. If so, the images in I g are merged into I r ' at step 1.7. At step 1.8, the most relevant image I t for the segment p t is identified, and I t is merged into I c ' at step 1.9. At step 1.10, the determination at step 1.6 can be ended. At step 1.11, the loop started at step 1.4 can be ended. Finally, the final I c ' including the selected image sequence or multiple images for the sentence S can be returned at step 1.12.

[0082] It should be understood that each of the steps in the processing shown in Table 1 is exemplary, which is intended to illustrate various exemplary processing procedures of embodiments of the present disclosure. According to specific application scenarios and requirements, any form of change can be made to these steps, and such changes will be covered by embodiments of the present disclosure.

[0083] According to embodiments of the present disclosure, after retrieving the image sequence corresponding to the text, the flexibility, relevance, visual consistency, etc. of the storyboard can be further enhanced by performing rendering processing on the image sequence. The rendering processing can include various different specific processing, for example, including at least one of relevant image area preservation processing, style unification processing, and element replacement processing.

[0084] Relevant image region preservation processing

[0085] Figure 5 An exemplary relevant image area preservation processing 500 according to an embodiment is shown.

[0086] It is assumed that the corresponding image sequence 520 has been retrieved for the text 510. Taking one sentence 512 in the text 510 as an example, at least one image 522 corresponding to the sentence 512 is included in the image sequence 520. The image 522 can include multiple image areas 530.

[0087] According to the process 500, a localization image region to which each word in the sentence 512 is localized can be identified from the plurality of image regions 530 included in the image 522. As previously described, each word in the sentence can be localized to a corresponding localization image region in the image by a dense visual semantic matching process.

[0088] At 540, it can be determined whether the identified localization image region is associated with a scene or an object. Here, a localization image region associated with a scene can refer to a portion of the image that involves a background or the like, and a localization image region associated with an object can refer to a portion of the image that involves a character or other object that is highlighted in the sentence or the like.

[0089] In one implementation, it can be determined whether the localization image region is associated with a scene or an object by determining whether the word corresponding to the localization image region describes a scene or an object. For example, if the word corresponding to a localization image region is “forest,” it can be determined that the localization image region is associated with a scene, and if the word corresponding to a localization image region is “boy” or “bread,” it can be determined that the localization image region is associated with an object. A scene word library including words describing scenes, an object word library including words describing objects, or the like can be created in advance, and whether a word in the sentence describes a scene or an object can be determined by semantically comparing the word in the sentence with the words in the word library.

[0090] In another implementation, it can be determined whether the localization image region is associated with a scene or an object by employing a Mask R-CNN model. In some cases, the Mask R-CNN model can be trained to generate an object segmentation mask or an object edge mask in the image specifically for an object. Whether the localization image region is associated with a scene or an object can be determined by comparing the localization image region with the object segmentation mask generated by the Mask R-CNN model. For example, if an overlap between the localization image region and the object segmentation mask is below a predetermined threshold, it can be determined that the localization image region is associated with a scene. Otherwise, it can be determined that the localization image region is associated with an object.

[0091] If the localized image region is determined to be associated with an object at 540, the localized image region can optionally be completed at 542. In some cases, although the localized image region determined through the dense visual semantic matching process has a correlation with the text in the sentence, the localized image region can not accurately cover the entire associated object due to the bottom-up attention network not being specifically designed to achieve high segmentation quality. For example, assuming the object is "boy", the localized image region can only include the body of the boy and not the head, etc. Mask R-CNN models can achieve higher object segmentation quality, e.g., can identify a complete region or edge corresponding to an object from an image. Thus, the localized image region can be completed using the object segmentation mask generated by the Mask R-CNN model through, e.g., a heuristic algorithm, etc. For example, the Mask R-CNN model can generate an object segmentation mask that completely covers the graph of the "boy" object, which can be used to complete the missed image portion of the localized image region. The completed localized image region can be retained as the relevant image region associated with the word in the sentence at 550.

[0092] If the localized image region is determined to be associated with a scene at 540, the localized image region can optionally be completed at 544. Similar to the discussion above for the localized image region associated with an object, the localized image region associated with a scene can also not accurately cover the entire associated scene. For example, assuming the scene is "forest", the localized image region can only include a portion of the forest in the image and not another portion, etc. In one implementation, Mask R-CNN models can also be used to complete the localized image region associated with a scene. In this case, the Mask R-CNN model can be extended to be trained to identify a scene segmentation mask from an image that defines a complete region or edge corresponding to a certain scene. The scene segmentation mask can be used to complete the missed image portion of the localized image region associated with a scene. The completed localized image region can then be retained as the relevant image region associated with the word in the sentence at 550.

[0093] The plurality of image regions 530 can also include image regions that are not localized by any word in the sentence 512, i.e., non-localized image regions 534. These non-localized image regions 534 can be removed from the image 522 as non-relevant image regions at 560.

[0094] It should be appreciated that the above-described process 500 is exemplary and can be changed in any form according to specific application scenarios and requirements. For example, the completion operation at 542 and / or the completion operation at 544 can be omitted, and the located image region 532 is directly reserved as a relevant image region at 550.

[0095] Figure 6 An example of performing a relevant image region reservation process according to an embodiment is shown. Image (a) is an original image retrieved corresponding to the sentence "a small boy eating bread". Image (b) shows located image regions 602 and 604 in image (a) located by dense visual semantic matching, where located image region 602 corresponds to the word "bread" in the sentence, and located image region 604 corresponds to the word "small boy" in the sentence. As can be seen from the figure, located image region 604 only includes the body part of the object "small boy", and misses the head part of the small boy. Image (c) shows the graphics of multiple objects identified in image (a) using object segmentation masks generated by a Mask R-CNN model, such as the graphics about the object "small boy", the graphics about the object "bread", the graphics about the object "table", etc. The object segmentation mask corresponding to the object "small boy" completely covers the object "small boy". In addition, image (c) can also include the object "table" which is not related to any word in the sentence. Image (d) shows the result of completing located image region 604 in image (b) using the object segmentation mask corresponding to the object "small boy" in image (c), where the completed located image region covers both the body part and the head part of the object "small boy". Since image (d) is completed on the basis of image (b) using the object segmentation mask in image (c), image (d) not only maintains the relevance to the sentence, but also guarantees the visual integrity of the object presented.

[0096] Style unification processing

[0097] The style unification process can unify different styles of multiple images in an image sequence. For example, multiple images that can have different styles are unified into the same desired style. The style unification process can be performed by any existing image style transfer technique. For example, a convolutional neural network model for converting images to a specific style can be pre-trained. The model can be used to convert or unify multiple images in an image sequence to the specific style.

[0098] Element replacement processing

[0099] The element replacement process can include at least one of a role image element replacement process, a scene image element replacement process, etc.

[0100] Figure 7 An exemplary character image element replacement process 700 according to an embodiment is shown.

[0101] Suppose a corresponding image sequence 720 has been retrieved for the text 710. The text 710 includes at least one character 712, e.g., "boy". At least one character image element 724 corresponding to the character 712 is included in an image 722 of the image sequence 720. The character image element 724 can be an image region of the image 722 that is about the boy.

[0102] A target character image 730 about the character 712 "boy" can be pre-designed. The target character image 730 can be a cartoon-type boy image, for example. An image library 732 based on the target character image 730 can be pre-created. A plurality of candidate image elements created using the target character image 730 can be included in the image library 732. Among the plurality of candidate image elements, the target character image 730 can have different poses, including actions, expressions, perspectives, etc. Any graphic design technique can be employed to create the candidate image elements. For example, a three-dimensional (3D) target character image can be first designed, then various different poses can be imposed on the target character image through any 3D design software, and finally the 3D design results can be mapped as 2D images to form the candidate image elements based on the target character image.

[0103] At 740, a pose matching can be performed between the character image element 724 and the candidate image elements in the image library 732. For example, a pose of the character 712 can be first identified from the character image element 724, and then a candidate image element having the same or similar pose can be searched from the image library 732 according to the identified pose. The searched candidate image element can be a target character image element 742. In this way, the target character image element 742 corresponding to the character image element 724 can be selected based on the pose matching.

[0104] At 750, the character image element 724 can be replaced by the target character image element 742 in the image 722.

[0105] By repeating the above process for all images and all characters in the image sequence 720, the character image elements involving the same character in the image sequence 720 can all be replaced by target character image elements based on the target character image. Thus, the consistency of the character image in the image sequence 720 is achieved.

[0106] It should be appreciated that the above-described process 700 is exemplary and can be changed in any form according to specific application scenarios and requirements. For example, instead of creating an image library based on target character images in advance, a target character image element based on the target character image can be created based on the pose of the character after the pose of the character is identified from the character image element 724. The creation of the target character image element can be in a similar manner as discussed above for creating the candidate image elements in the database 732.

[0107] Figure 8 An exemplary scene image element replacement process 800 according to an embodiment is shown.

[0108] It is assumed that a corresponding image sequence 820 has been retrieved for the text 810. At least one scene 812 can be identified from the text 810. For example, the text 810 can include the words "wild wind", "forest", etc., which describe a scene such as "forest in wild wind".

[0109] At least one scene image element 824 associated with the scene 812 can be identified in an image 822 of the image sequence 820. The scene image element 824 can be an image region in the image 822, for example, about the forest.

[0110] A database 830 can be created in advance, which includes various candidate target scene image elements, for example, with a uniform style. In one implementation, the style can match the target character image involved in the scene 812. Figure 7

[0111] At 840, a target scene image element 842 associated with the scene 812 can be selected from the image library 830. For example, the target scene image element 842 can exhibit a scene of trees swaying in the wind.

[0112] At 850, the scene image element 824 can be replaced by the target scene image element 842 in the image 822.

[0113] By repeatedly performing the above process for all images in the image sequence 820, all scene image elements in the image sequence 820 can be replaced by target scene image elements. Thus, the image sequence after the scene image element replacement can more fully exhibit the scene described in the sentence, and can have scene image elements with a uniform style.

[0114] It should be appreciated that the above-described process 800 is exemplary and can be changed in any form according to specific application scenarios and requirements.

[0115] Figure 9 ​An example of retrieving and rendering an image sequence according to an embodiment is shown. In this example, only one image is retrieved for each sentence in the text.

[0116] It is assumed that a storyboard is to be generated for the text 910. The text 910 may, for example, include sentences 912, 914 and 916.

[0117] The image sequence 920 corresponding to the multiple sentences in the text 910 can be retrieved by, for example, the text-to-image retriever 110. Figure 1 The image sequence 920 may, for example, include an image 922 corresponding to the sentence 912, an image 924 corresponding to the sentence 914, and an image 926 corresponding to the sentence 916.

[0118] Further rendering processing can be performed on the retrieved image sequence 920.

[0119] After the relevant image region preservation processing, an image sequence 930 can be obtained. The image sequence 930 includes images 932, 934 and 936 obtained from the images 922, 924 and 926 respectively. Take the image 932 as an example, the relevant image regions located by the words "pie", "dot on sesame" and the like in the sentence 912 are preserved in the image 932, while other irrelevant image regions are removed.

[0120] Then, style uniform processing can be performed on the image sequence 930 after the relevant image region preservation processing to obtain an image sequence 940. The image sequence 940 includes images 942, 944 and 946 obtained from the images 932, 934 and 936 respectively. The multiple images in the image sequence 940 have a uniform style, such as color tone, line, and the like.

[0121] The image sequence 940 after the style uniform processing can be further used to form a storyboard (not shown).

[0122] It should be understood that although not shown, additional processing can also be applied to the example in Figure 9 For example, element replacement processing and the like can be further performed on the image sequence 940. In addition, it should be understood that all the images in Figure 9 are exemplary, and these images can represent images from any type of source. For example, the images in the retrieved image sequence 920 can represent actual shot pictures from a movie.

[0123] Figure 10A and Figure 10B An example of retrieving and rendering an image sequence according to an embodiment is shown. In this example, more than one image is retrieved for each sentence in the text, so that the content in each sentence can be more fully presented. It is still assumed that a storyboard is to be generated for the text 910. Figure 9The text 910 in the middle generates a storyboard.

[0124] It can be done through, for example Figure 1 The text-to-image retrieval unit 110 retrieves image sequences 1010 corresponding to multiple statements in the text 910. Multiple images can be retrieved for each statement using the one-to-many image retrieval method discussed above. For example, image sequence 1010 may include an image set 1012 corresponding to statement 912, which includes images 1012-1, 1012-2, 1012-3, and 1012-4 corresponding to multiple segments in statement 912. Image sequence 1010 may include an image set 1014 corresponding to statement 914, which includes images 1014-1, 1014-2, and 1014-3 corresponding to multiple segments in statement 914. Image sequence 1010 may include an image set 1016 corresponding to statement 916, which includes images 1016-1 and 1016-2 corresponding to multiple segments in statement 916. Taking image set 1016 as an example, in image set 1016, image 1016-1 can correspond to the sentence "The little boy's parents found him" in sentence 916, while image 1016-2 can correspond to the sentence "He ate a really delicious pancake" in sentence 916.

[0125] The retrieved image sequence 1010 can be further rendered.

[0126] After relevant image region preservation processing, image sequence 1020 is obtained. Image sequence 1020 includes image sets 1022, 1024, and 1026 obtained from image sets 1012, 1014, and 1016, respectively. For example, image set 1022 includes images 1022-1, 1022-2, 1022-3, and 1022-4 obtained from images 1012-1, 1012-2, 1012-3, and 1012-4, respectively; image set 1024 includes images 1024-1, 1024-2, and 1024-3 obtained from images 1014-1, 1014-2, and 1014-3, respectively; and image set 1026 includes images 1026-1 and 1026-2 obtained from images 1016-1 and 1016-2, respectively.

[0127] Then, the image sequence 1030 that has undergone the style uniformization processing can be subjected to an element replacement processing to obtain an image sequence 1040. The image sequence 1040 includes image sets 1042, 1044 and 1046 obtained from the image sets 1032, 1034 and 1036, respectively. For example, the image set 1042 includes images 1042-1, 1042-2, 1042-3 and 1042-4 obtained from the images 1032-1, 1032-2, 1032-3 and 1032-4, respectively, the image set 1044 includes images 1044-1, 1044-2 and 1044-3 obtained from the images 1034-1, 1034-2 and 1034-3, respectively, and the image set 1046 includes images 1046-1 and 1046-2 obtained from the images 1036-1 and 1036-2, respectively. In the image sequence 1040, all the character image elements corresponding to the character "boy" in the text 910 are replaced with target character image elements based on a target character image, which is a cartoon-type boy image. In addition, in the image sequence 1040, the scene image elements in the image sequence 1030 are also replaced with target scene image elements corresponding to the scenes described in the text 910, respectively.

[0128] Then, the image sequence 1030 that has undergone the style uniformization processing can be subjected to an element replacement processing to obtain an image sequence 1040. The image sequence 1040 includes image sets 1042, 1044 and 1046 obtained from the image sets 1032, 1034 and 1036, respectively. For example, the image set 1042 includes images 1042-1, 1042-2, 1042-3 and 1042-4 obtained from the images 1032-1, 1032-2, 1032-3 and 1032-4, respectively, the image set 1044 includes images 1044-1, 1044-2 and 1044-3 obtained from the images 1034-1, 1034-2 and 1034-3, respectively, and the image set 1046 includes images 1046-1 and 1046-2 obtained from the images 1036-1 and 1036-2, respectively. In the image sequence 1040, all the character image elements corresponding to the character "boy" in the text 910 are replaced with target character image elements based on a target character image, which is a cartoon-type boy image. In addition, in the image sequence 1040, the scene image elements in the image sequence 1030 are also replaced with target scene image elements corresponding to the scenes described in the text 910, respectively.

[0129] The image sequence 1040 that has undergone the element replacement processing can be further used to form a storyboard (not shown).

[0130] It should be understood that, Figure 10A and Figure 10B all the images in Figure 10A and Figure 10BThe processing in the above-described manner can be changed in any way. For example, the style unification processing can be omitted, and the element replacement processing can be directly performed on the image sequence 1020 that has undergone the relevant image region reservation processing, and the like.

[0131] Figure 11 A flowchart illustrating an example method 1100 for automatically generating a storyboard according to an embodiment is shown.

[0132] At 1110, a text including a plurality of sentences can be obtained.

[0133] At 1120, an image sequence corresponding to the plurality of sentences can be retrieved.

[0134] At 1130, the image sequence can be rendered to obtain a rendered image sequence.

[0135] At 1140, a storyboard including the rendered image sequence can be provided.

[0136] In an implementation, the retrieving can include, for each sentence in the plurality of sentences: generating a word representation sequence of the sentence by context-aware encoding; and selecting at least one image corresponding to the sentence based on the word representation sequence by dense visual semantic matching.

[0137] The context-aware encoding can be based on a hierarchical attention mechanism. The hierarchical attention mechanism can include: a first attention level for selecting a cross-sentence context in the plurality of sentences for each word in the sentence; and a second attention level for performing weighting for the cross-sentence context selected for each word in the sentence.

[0138] The dense visual semantic matching can include: localizing each word in the sentence to a localization image region in the at least one image.

[0139] The selecting can include: dividing the sentence into a plurality of sentence segments; obtaining a plurality of sentence segment representations corresponding to the plurality of sentence segments, respectively, based on the word representation sequence; and selecting a plurality of images corresponding to the plurality of sentence segments, respectively, based on the plurality of sentence segment representations by the dense visual semantic matching.

[0140] In an implementation, the rendering can include at least one of: performing a relevant image region reservation processing on each image in the image sequence, the relevant image region reservation processing including reserving a relevant image region associated with a word in a sentence corresponding to the image and / or removing a non-relevant image region; performing a style unification processing on the image sequence; and performing an element replacement processing on the image sequence.

[0141] The retaining relevant image region can include: identifying a locating image region in the image where each term in the sentence is located; completing the locating image region by a mask region convolutional neural network model; and retaining the completed locating image region as a relevant image region associated with the term. The locating image region can be associated with a scene or an object.

[0142] The removing irrelevant image region can include: removing an image region from the image that is not located by any term in the sentence.

[0143] The performing element replacement processing can include: identifying at least one character image element associated with a same character in the plurality of sentences from the image sequence; selecting at least one target character image element corresponding to the at least one character image element from an image library created based on a target character image of the character based on pose matching; and replacing the at least one character image element in the image sequence with the at least one target character image element.

[0144] The performing element replacement processing can include: identifying at least one scene from the plurality of sentences; identifying at least one scene image element associated with the at least one scene from the image sequence; selecting at least one target scene image element associated with the at least one scene from a pre-created image library; and replacing the at least one scene image element in the image sequence with the at least one target scene image element.

[0145] It should be understood that the method 1100 can further include any steps / processes for automatically generating a storyboard according to the above disclosed embodiments.

[0146] Figure 12 An exemplary apparatus 1200 for automatically generating a storyboard according to an embodiment is shown.

[0147] The apparatus 1200 can include: a text obtaining module 1210 configured to obtain a text including a plurality of sentences; an image sequence retrieving module 1220 configured to retrieve an image sequence corresponding to the plurality of sentences; an image sequence rendering module 1230 configured to render the image sequence to obtain a rendered image sequence; and a storyboard providing module 1240 configured to provide a storyboard including the rendered image sequence.

[0148] In addition, the apparatus 1200 can further include any other module configured to perform any steps / processes in the method for automatically generating a storyboard according to the above disclosed embodiments.

[0149] Figure 13An exemplary apparatus 1300 for automatically generating a storyboard according to an embodiment is shown.

[0150] The apparatus 1300 can include at least one processor 1310 and a memory 1320 storing computer executable instructions. When executing the computer executable instructions, the processor 1310 can obtain a text including a plurality of sentences, retrieve an image sequence corresponding to the plurality of sentences, render the image sequence to obtain a rendered image sequence, and provide a storyboard including the rendered image sequence. In addition, the processor 1310 can also perform any steps / processes for automatically generating a storyboard according to the above disclosed embodiments.

[0151] Embodiments of the present disclosure also provide an apparatus for automatically generating a storyboard, comprising: a text-to-image retriever configured to retrieve an image sequence corresponding to a plurality of sentences in a text; and a storyboard creator configured to render the image sequence to obtain a rendered image sequence, and form a storyboard based on the rendered image sequence.

[0152] In an implementation, the text-to-image retriever can be configured to, for each sentence in the plurality of sentences: generate a word representation sequence of the sentence by context-aware encoding; and select at least one image corresponding to the sentence based on the word representation sequence by dense visual semantic matching.

[0153] The context-aware encoding can be based on a hierarchical attention mechanism. The hierarchical attention mechanism can include: a first attention level configured to select a cross-sentence context in the plurality of sentences for each word in the sentence; and a second attention level configured to perform weighting for the cross-sentence context selected for each word in the sentence.

[0154] The dense visual semantic matching can include: localizing each word in the sentence to a localizing image region in the at least one image.

[0155] The selecting can include: dividing the sentence into a plurality of sentence segments; obtaining a plurality of sentence segment representations corresponding to the plurality of sentence segments respectively based on the word representation sequence; and selecting a plurality of images corresponding to the plurality of sentence segments respectively based on the plurality of sentence segment representations by the dense visual semantic matching.

[0156] In an implementation, the storyboard creator can be configured to render the image sequence by at least one of: performing a relevant image region preserving process on each image in the image sequence, the relevant image region preserving process comprising preserving relevant image regions associated with words in a sentence corresponding to the image and / or removing irrelevant image regions; performing a style unifying process on the image sequence; and performing an element replacing process on the image sequence.

[0157] Further, the text-to-image retriever and the storyboard creator can also perform any of the steps / processes for automatically generating a storyboard according to the above-described embodiments of the present disclosure.

[0158] Embodiments of the present disclosure can be implemented in a non-transitory computer readable medium. The non-transitory computer readable medium can include instructions that, when executed, cause one or more processors to perform any of the operations of the method for automatically generating a storyboard according to the above-described embodiments of the present disclosure.

[0159] It should be understood that all of the operations in the above-described method are merely exemplary, and the present disclosure is not limited to any of the operations in the method or the order of the operations, but should cover all other equivalent transformations under the same or similar concepts.

[0160] It should also be understood that all of the modules in the above-described apparatus can be implemented in various ways. The modules can be implemented as hardware, software, or a combination thereof. Further, any of the modules can be further divided functionally into sub-modules or combined together.

[0161] Processors have been described in connection with various apparatus and methods. These processors can be implemented using electronic hardware, computer software, or any combination thereof. Whether these processors are implemented as hardware or software depends on the particular application and general design constraints imposed on the system. As an example, the processors given in the present disclosure, any portion of the processors, or any combination of the processors can be implemented as a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, a gate logic, a discrete hardware circuit, and other suitable processing components configured to perform the various functions described in the present disclosure. The functions of the processors given in the present disclosure, any portion of the processors, or any combination of the processors can be implemented as software executed by a microprocessor, a microcontroller, a DSP, or other suitable platform.

[0162] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, threads of execution, procedures, functions, etc. The software can reside on a computer-readable medium. The computer-readable medium can include, for example, memory such as a magnetic storage device (e.g., hard disk); optical storage devices; flash memory devices; random access memory (RAM); read only memory (ROM); programmable ROM (PROM); erasable PROM (EPROM); electrically erasable PROM (EEPROM); registers; or a removable disk. Although memory is shown separate from the processors in the numerous aspects presented herein, the memory can be internal to the processors (e.g., cache or register).

[0163] This description is provided to enable any person skilled in the art to make and use the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects. Thus, the claims are not intended to be limited to the aspects presented herein. All structures and functions described herein can be combined in any combination. All references are incorporated by reference in their entirety.

Claims

1. A method for automatically generating a storyboard, comprising: obtaining a text comprising a plurality of sentences; retrieving a sequence of images corresponding to the plurality of sentences, wherein the retrieving comprises, for each sentence in the plurality of sentences: generating a sequence of word representations for the sentence by context-aware encoding; dividing the sentence into a plurality of segments; obtaining a plurality of segment representations corresponding to the plurality of segments respectively based on the sequence of word representations; and selecting a plurality of images corresponding to the sentence based on the plurality of segment representations by dense visual semantic matching, wherein the selecting comprises, for each segment: selecting a set of images most relevant to the segment; determining that the set of images has not been retrieved for a preceding segment of the sentence; and in response to determining that the set of images has not been retrieved for the preceding segment of the sentence, incorporating an image most relevant to the segment into the plurality of images corresponding to the sentence; rendering the sequence of images to obtain a rendered sequence of images; and providing a storyboard comprising the rendered sequence of images.

2. The method of claim 1, wherein, the context-aware encoding is based on a hierarchical attention mechanism.

3. The method of claim 2, wherein, the hierarchical attention mechanism comprises: a first attention level for selecting a cross-sentence context among the plurality of sentences for each word in the sentence; and a second attention level for performing weighting for the cross-sentence context selected for each word in the sentence.

4. The method of claim 1, wherein, the dense visual semantic matching comprises: localizing each word in the sentence to a localized image region in the plurality of images.

5. The method of claim 1, wherein, the rendering comprises: identifying, from a plurality of image regions comprised by an image in the sequence of images, a localized image region localized to a word in a sentence corresponding to the image; determining that the identified localized image region is associated with a scene or an object; and in response to determining that the identified localized image region is associated with a scene or an object, completing the identified localized image region.

6. The method of claim 1, wherein, the rendering comprises at least one of: performing a relevant image region preservation processing on each image in the sequence of images, the relevant image region preservation processing comprising preserving a relevant image region associated with a word in a sentence corresponding to the image and / or removing a non-relevant image region; performing a style uniformization processing on the sequence of images; and performing an element replacement processing on the sequence of images. the preserving a relevant image region associated with a word in a sentence corresponding to the image comprises:

7. The method of claim 6, wherein, identifying a localized image region in the image localized to each word in the sentence; completing the localized image region by a masked region convolutional neural network model; and preserving the completed localized image region as the relevant image region associated with the word. the localized image region is associated with a scene or an object.

8. The method of claim 7, wherein, the removing a non-relevant image region comprises:

9. The method of claim 6, wherein, removing, from the image, an image region not localized to any word in the sentence. the performing an element replacement processing comprises:

10. The method of claim 6, wherein, identifying, from the sequence of images, at least one character image element associated with a same character in the plurality of sentences; ​ based on the pose matching, selecting, from an image library created based on target character images of the character, at least one target character image element corresponding to the at least one character image element; and replacing the at least one character image element in the image sequence with the at least one target character image element.

11. The method of claim 6, wherein, The performing element replacement processing comprises: identifying, from the plurality of sentences, at least one scene; identifying, from the image sequence, at least one scene image element associated with the at least one scene; selecting, from a pre-created image library, at least one target scene image element associated with the at least one scene; and replacing the at least one scene image element in the image sequence with the at least one target scene image element.

12. An apparatus for automatically generating a storyboard, comprising: a text-to-image retriever configured to retrieve an image sequence corresponding to a plurality of sentences in a text; and a storyboard creator configured to render the image sequence to obtain a rendered image sequence, and form a storyboard based on the rendered image sequence, wherein the text-to-image retriever is further configured to, for each sentence in the plurality of sentences: generate a word representation sequence of the sentence by context-aware encoding; divide the sentence into a plurality of segments; obtain a plurality of segment representations corresponding to the plurality of segments respectively based on the word representation sequence; and select a plurality of images corresponding to the sentence based on the plurality of segment representations by dense visual semantic matching, wherein the selecting comprises, for each segment: selecting a set of images most relevant to the segment; determining that the set of images has not been retrieved for a preceding segment of the sentence; and in response to determining that the set of images has not been retrieved for the preceding segment of the sentence, incorporating the image most relevant to the segment into the plurality of images corresponding to the sentence.

13. The apparatus of claim 12, wherein, The context-aware encoding is based on a hierarchical attention mechanism.

14. The apparatus of claim 13, wherein, The hierarchical attention mechanism comprises: a first attention level configured to select, for each word in the sentence, a cross-sentence context among the plurality of sentences; and a second attention level configured to perform weighting for the cross-sentence context selected for each word in the sentence.

15. The apparatus of claim 12, wherein, The dense visual semantic matching comprises: localizing each word in the sentence to a localized image region in the plurality of images.

16. The apparatus of claim 12, wherein, The storyboard creator is configured to render the image sequence by: identifying, from a plurality of image regions comprised in an image in the image sequence, a localized image region localized by a word in a sentence corresponding to the image; determining that the identified localized image region is associated with a scene or an object; and in response to determining that the identified localized image region is associated with a scene or an object, completing the identified localized image region. The storyboard creator is configured to render the image sequence by at least one of:

17. The apparatus of claim 12, wherein, ​ performing, for each image in the sequence of images, a relevant image region preservation process, the relevant image region preservation process including preserving relevant image regions associated with words in a sentence corresponding to the image and / or removing irrelevant image regions; performing a style unification process on the sequence of images; and performing an element replacement process on the sequence of images.

18. An apparatus for automatically generating a storyboard, comprising: at least one processor; and a memory storing computer-executable instructions that, when executed, cause the at least one processor to: obtain a text comprising a plurality of sentences, retrieve a sequence of images corresponding to the plurality of sentences, wherein the retrieving includes, for each sentence in the plurality of sentences: generate a sequence of word representations of the sentence by contextualized encoding, divide the sentence into a plurality of segments, obtain a plurality of segment representations corresponding to the plurality of segments respectively based on the sequence of word representations, and select a plurality of images corresponding to the sentence based on the plurality of segment representations by dense visual semantic matching, wherein the selecting includes, for each segment: selecting a set of images most relevant to the segment, determining that the set of images has not been retrieved for a preceding segment of the sentence, and in response to determining that the set of images has not been retrieved for the preceding segment of the sentence, incorporating an image most relevant to the segment into the plurality of images corresponding to the sentence, render the sequence of images to obtain a rendered sequence of images, and provide a storyboard comprising the rendered sequence of images.

Images