Method and electronic device for creating continuity in story

Abstract

Embodiments herein disclose a method and an electronic device (100) for creating continuity in a story. The method includes receiving, by the electronic device (100) as input, at least one first image and at least one second image, determining a plurality of parameters associated with the first image and a plurality of parameters associated with the second image. The plurality of parameters associated with the first image and the second image include scene elements, actions of the scene elements, and topics formed by the scene elements. The method further includes generating a graphical representation to connect the first image with the second image, and displaying a story including the first image, the second image, and the graphical representation generated between the first image and the second image.

Description

Technical Field

[0001] The present disclosure relates to electronic devices, and more particularly, to a method and electronic device for creating continuity in a story. Background Art

[0002] In general, photo stories make it easier to share photos and videos with friends and family. Conventionally, photos are grouped into photo stories based on analysis of metadata associated with the photos by selecting a set of images based on a predefined strategy. In addition, multiple personalized storylines are generated from a given set of photos. Conventional systems and methods automatically extract keywords related to themes from users' natural language comments on photos and videos. Here, "theme" refers to concepts that define and describe the content of photos and videos, such as pets, natural landscapes, palaces, and places. The method adopts a deep learning algorithm (recurrent neural network (RNN)) to recognize implicit patterns in sequence data.

[0003] In addition, conventional systems and methods do not understand the theme of the story and do not evaluate the image features of each photo for the theme of the story. In addition, evaluating the image features of each photo for the theme of the story is important for creating continuity in the photo story.

[0004] Conventional methods and systems create, manage, and share photo stories by selecting a set of photo story design templates for each of the different photo stories based on an analysis of the photos and metadata associated with the photos grouped into the different photo stories. However, the conventional methods and systems focus on grouping the photos into stories based on an analysis of the metadata associated with the photos, but do not predict bridging events between the photos in the stories, nor do they complete the story visualization by adding generated scenes between the photos that fit the story theme.

[0005] It would therefore be desirable to address the above-mentioned or other disadvantages or at least provide a useful alternative. Summary of the invention

[0006] Therefore, embodiments of the present invention disclose a method for creating continuity in a story through an electronic device. The method includes receiving a first image and a second image as input. The method includes determining a plurality of parameters associated with the first image and a plurality of parameters associated with the second image. The method also includes generating a graphical representation based on the plurality of parameters associated with the first image and the plurality of parameters associated with the second image to connect the first image with the second image. In addition, the method includes displaying a story including the first image, the second image, and the graphical representation generated between the first image and the second image.

[0007] In an embodiment, the plurality of parameters include scene elements in the first image and the second image, motions of the scene elements in the first image and the second image, and a theme formed by the scene elements in the first image and the second image.

[0008] In an embodiment, a graphical representation is generated to connect a first image with a second image by predicting a bridging event connecting the first image with the second image based on a plurality of parameters associated with the first image and a plurality of parameters associated with the second image, and generating a graphical representation of the bridging event to connect the first image with the second image.

[0009] In an embodiment, a bridging event connecting a first image with a second image is predicted by the following steps: creating a text summary of the first image and the second image based on a plurality of parameters associated with the first image and a plurality of parameters associated with the second image; generating a first image feature for the first image based on the text summary created for the first image; generating a second image feature for the second image based on the text summary created for the second image; and predicting a bridging event for connecting the first image and the second image based on the first image feature generated for the first image and the second image feature generated for the second image.

[0010] In an embodiment, a graphical representation of a bridging event is generated to connect a first image with a second image by the following steps: comparing the first image and the second image based on a theme formed by scene elements in the first image and a theme formed by scene elements in the second image; determining that an image relationship distance between the first image and the second image is less than a first threshold; and when the image relationship distance between the first image and the second image is less than the first threshold, generating a graphical representation of a bridging event to connect the first image with the second image.

[0011] In an embodiment, the method includes: obtaining multiple images; identifying features of an object available in each of the multiple images; determining a similarity score for grouping images having similar features of the object in the multiple images into spans; generating multiple spans of the multiple images based on the similarity scores between the images in the multiple images; determining a visual scene distance / collaborative visual scene distance between multiple spans of the multiple images; determining that the visual scene distance / collaborative visual scene distance between the multiple spans of the multiple images is less than a second threshold; and when the visual scene distance / collaborative visual scene distance between the multiple spans of the multiple images is less than the second threshold, generating a graphical representation of a bridging event to connect the multiple spans of the multiple images.

[0012] In an embodiment, the step of generating multiple spans of multiple images based on similarity scores between images in a plurality of images includes: determining a trajectory of an object available in a first image in the plurality of images and a trajectory of an object available in a second image in the plurality of images; identifying features of an object available in the first image in the plurality of images and features of an object available in the second image based on the determined trajectory of the object available in the first image in the plurality of images and the second image in the plurality of images; determining that a similarity score between a first image in the plurality of images and a second image in the plurality of images is higher than a third threshold; and generating multiple spans of the plurality of images based on the similarity score between the first image in the plurality of images and the second image in the plurality of images.

[0013] Therefore, an embodiment of the present invention discloses an electronic device for creating continuity in a story. The electronic device includes a memory, a processor coupled to the memory, a communicator coupled to the memory and the processor, and a story management controller coupled to the memory, the processor, and the communicator. The story management controller is configured to receive a first image and a second image as input, and determine a plurality of parameters associated with the first image and a plurality of parameters associated with the second image. The story management controller is also configured to generate a graphical representation based on the plurality of parameters associated with the first image and the plurality of parameters associated with the second image to connect the first image with the second image, and display a story including the first image, the second image, and the graphical representation generated between the first image and the second image.

[0014] When considered in conjunction with the following description and the accompanying drawings, these and other aspects of the embodiments herein will be better appreciated and understood. However, it should be understood that the following description, although indicating preferred embodiments and many specific details thereof, is intended to be illustrative and not limiting. Many changes and modifications may be made within the scope of the embodiments herein without departing from the present invention, and the embodiments herein include all of these modifications. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention is illustrated in the accompanying drawings, in which the same reference numerals indicate corresponding parts in the various figures throughout the drawings. The embodiments of the present invention will be better understood from the following description with reference to the accompanying drawings, in which: Figure 1 is a block diagram of an electronic device for creating continuity in a story according to embodiments disclosed herein; Figure 2 is a flow chart illustrating a method for creating continuity in a story through an electronic device according to an embodiment disclosed herein; Figure 3 is a block diagram of a system for creating continuity in a story according to embodiments disclosed herein; Figure 4is an example showing that images with gaps will be bridged to bring continuity in the story according to the prior art; Figure 5a and Figure 5b is an example showing a process for bridging a gap between two images according to embodiments disclosed herein; Figure 6 is an example showing a story visualization based on scene distance according to an embodiment disclosed herein; Figure 7 is a block diagram illustrating a multi-image story summary according to embodiments disclosed herein; and Figure 8 is a block diagram illustrating a process of creating a story-based text summary for a single image according to embodiments disclosed herein. DETAILED DESCRIPTION

[0016] The main purpose of the embodiments of this document is to provide a method and electronic device for creating continuity in a story. The method includes determining scene elements in two or more images, actions of the scene elements in the two or more images, and a theme formed by the scene elements in the two or more images, and predicting a bridging event that connects the themes of the two or more images.

[0017] Another object of embodiments herein is to generate a graphical representation of a bridging event connecting two or more images based on the subject matter of the two or more images, thereby creating continuity in the story.

[0018] Therefore, the proposed method creates a story from two or more images, wherein a graphical representation of a bridging event created from the two or more images is inserted between the two or more images to bridge the gap between the two or more images with respect to one or more of actions, events, characters, and scenes, thereby enhancing the user's story viewing experience by creating continuity in the story.

[0019] The embodiments of the present invention and their various features and advantageous details are explained more fully with reference to the non-limiting embodiments shown in the accompanying drawings and described in detail in the following description. The description of well-known components and processing techniques is omitted so as not to unnecessarily obscure the embodiments of the present invention. In addition, the various embodiments described herein are not necessarily mutually exclusive, because some embodiments can be combined with one or more other embodiments to form new embodiments. Unless otherwise stated, the term "or" used herein refers to a non-exclusive or. The examples used herein are intended only to facilitate understanding of the ways in which the embodiments of the present invention can be practiced, and also to enable those skilled in the art to practice the embodiments of the present invention. Therefore, the examples should not be interpreted as limiting the scope of the embodiments of the present invention.

[0020] As is conventional in the art, embodiments may be described and illustrated according to blocks that perform one or more functions described. These blocks, which may be referred to herein as units or modules, etc., are physically implemented by analog circuits or digital circuits (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.), and may optionally be driven by firmware. The circuit may be implemented, for example, in one or more semiconductor chips, or on a substrate support (such as a printed circuit board, etc.). The circuits constituting the blocks may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware that performs some functions of the blocks and a processor that performs other functions of the blocks. Without departing from the scope of the present disclosure, each block of the embodiment may be physically divided into two or more interacting and discrete blocks. Similarly, without departing from the scope of the present disclosure, the blocks of the embodiment may be physically combined into more complex blocks.

[0021] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. Therefore, in addition to those specifically set forth in the accompanying drawings, the present disclosure should also be interpreted as extending to any changes, equivalents and alternative forms. Although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are usually only used to distinguish one element from another element.

[0022] Therefore, embodiments of the present invention disclose a method for creating continuity in a story by an electronic device. The method includes receiving a first image and a second image as input. The method includes determining a plurality of parameters associated with the first image and a plurality of parameters associated with the second image. The plurality of parameters include scene elements in the first image and the second image, actions of the scene elements in the first image and the second image, and a theme formed by the scene elements in the first image and the second image. The method also includes generating a graphical representation based on the plurality of parameters associated with the first image and the plurality of parameters associated with the second image to connect the first image with the second image. In addition, the method includes displaying a story including the first image, the second image, and the graphical representation generated between the first image and the second image.

[0023] Therefore, an embodiment of the present invention discloses an electronic device for creating continuity in a story. The electronic device includes a memory, a processor coupled to the memory, a communicator coupled to the memory and the processor, and a story management controller coupled to the memory, the processor, and the communicator. The story management controller is configured to receive a first image and a second image as input, and determine a plurality of parameters associated with the first image and a plurality of parameters associated with the second image. The plurality of parameters include scene elements in the first image and the second image, actions of the scene elements in the first image and the second image, and a theme formed by the scene elements in the first image and the second image. The story management controller also generates a graphical representation based on the plurality of parameters associated with the first image and the plurality of parameters associated with the second image to connect the first image with the second image. In addition, the story management controller displays a story including the first image, the second image, and the graphical representation generated between the first image and the second image.

[0024] Conventional methods and systems create, manage, and share photo stories. The method receives photos and metadata associated with the photos from a user. The photos and metadata associated with the photos are analyzed, and in response, the photos are grouped into a plurality of different photo stories based on the analysis of the photos and metadata associated with the photos. A set of photo story design templates for each photo story in the different photo stories is selected based on the analysis of the photos and metadata associated with the photos grouped into the different photo stories. However, conventional methods and systems focus on grouping photos into stories based on the analysis of metadata associated with the photos, but do not predict bridging events between photos in the story, nor do they complete the story visualization by adding scenes generated between the photos that fit the story theme.

[0025] Conventional methods and systems extract keywords related to the theme from comments provided on photos. Here, themes are extracted from photos and videos, and related keywords are extracted from comments provided on these contents. However, conventional methods and systems do not understand the theme of the story and do not generate a theme understanding of the story. In addition, conventional methods and systems do not evaluate the image features of each photo with respect to the theme of the story to generate bridging events to connect the images in the story.

[0026] Unlike conventional methods and systems, the proposed method predicts a bridging event for connecting themes of two or more images, and generates a graphical representation of the bridging event based on the themes of the two or more images to connect the two or more images. In addition, the proposed method displays a story including a first image, a second image, and a graphical representation generated between the first image and the second image to create continuity in the story. Therefore, by creating continuity in the story, the gap between the first image and the second image regarding one or more of actions, events, characters, and scenes is bridged, and the user experience of story viewing is enhanced.

[0027] Referring now to the drawings, and more particularly to Figures 1 to 8 , wherein like reference numerals denote corresponding features throughout the drawings, there are shown preferred embodiments.

[0028] Figure 1 is a block diagram of an electronic device (100) for creating continuity in a story according to an embodiment disclosed herein. Figure 1 The electronic device (100) may be, but is not limited to, a laptop computer, a handheld computer, a desktop computer, a mobile phone, a smart phone, a personal digital assistant (PDA), a tablet computer, a wearable device, an Internet of Things (IoT) device, a virtual reality device, a foldable device, a flexible device, a display device, and an immersive system.

[0029] In an embodiment, an electronic device (100) includes a memory (110), a processor (120), a communicator (130), a story management controller (140), and a display (150).

[0030] The memory (110) is configured to store a plurality of images received as input. The memory (110) may include a non-volatile storage element. Examples of such non-volatile storage elements may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or a form of an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory. In addition, in some examples, the memory (110) may be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not implemented in a carrier wave or propagating signal. However, the term "non-transitory" should not be interpreted as the memory (110) being non-removable. In some examples, the memory (110) is configured to store a larger amount of information. In a specific example, the non-transitory storage medium may store data that may change over time (e.g., in a random access memory (RAM)).

[0031] The processor (120) may include one or more processors. The one or more processors may be general-purpose processors (such as a central processing unit (CPU), an application processor (AP), etc.), a graphics processing unit (such as a graphics processing unit (GPU), a visual processing unit (VPU)), and / or an AI-specific processor (such as a neural processing unit (NPU)). The processor (120) may include multiple cores and be configured to analyze multiple images stored in the memory (110).

[0032] In an embodiment, the communicator (130) includes electronic circuits dedicated to implementing standard wired or wireless communication. The communicator (130) is configured to communicate internally between internal hardware components of the electronic device (100) and to communicate with external devices via one or more networks.

[0033] In an embodiment, the story management controller (140) includes an image receiver (141), a story theme analyzer (142), a scene analyzer (143), a relationship proximity detector (144), an image sequencer (145), a bridging event predictor (146), and a text summarizer (147).

[0034] In an embodiment, the image receiver (141) of the story management controller (140) is configured to receive an image of a story as input.

[0035] In an embodiment, the story theme analyzer (142) of the story management controller (140) is configured to determine parameters associated with the image. The parameters include, but are not limited to, scene elements in the image (such as, for example, but not limited to, players, fitness coaches / trainees), actions of scene elements in the image (such as, for example, but not limited to, games, fitness training), and themes formed by the scene elements in the image. Themes include, but are not limited to, games, fitness trips, birthday celebrations, travel itineraries formed by the scene elements in the image. The story theme analyzer (142) is also configured to summarize the entire story into a text format based on the determination of the parameters associated with the image.

[0036] In an embodiment, the scene analyzer (143) is configured to analyze and predict a scene connecting one image to another image based on parameters associated with the images.

[0037] In an embodiment, the relational proximity detector (144) is configured to detect a relational proximity distance between consecutive images including objects (such as, for example but not limited to, a ball, a net, a fitness tool, a building, an animal, a vehicle, etc.), which may refer to "image relationship distance", "visual scene distance" or "collaborative visual scene distance" in the claims of the present disclosure.

[0038] In an embodiment, the image sequencer (145) is configured to identify features and similarities of objects present in the images. The image sequencer (145) is configured to arrange and group images having similarity scores above a threshold in order based on a relationship proximity distance between the images and features and similarities of objects present in the images.

[0039] In an embodiment, the bridging event predictor (146) is configured to predict a bridging event to connect the images based on parameters associated with the images. More particularly, the bridging event predictor (146) is configured to predict a bridging event to connect the images based on a theme formed by scene elements in different images.

[0040] In an embodiment, the text summarizer (147) is configured to receive a plurality of images as input and generate a text summary of the plurality of images that captures the most important elements, themes, scenes, etc. In addition, the text summarizer (147) creates an understanding of how the images fit into the story to generate a graphical representation of bridging events to connect the images.

[0041] The story management controller (140) is implemented by processing circuits (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.), and may optionally be driven by firmware. The circuits may be implemented, for example, in one or more semiconductor chips, or on a substrate support (such as a printed circuit board, etc.).

[0042] At least one of the plurality of modules / components of the graphics management controller (140) may be implemented by an AI model. Functions associated with the AI ​​model may be performed by a memory (110) and a processor (120). One or more processors control processing of input data according to predefined operating rules or AI models stored in non-volatile memory and volatile memory. The predefined operating rules or AI models are provided by training or learning.

[0043] Here, providing by learning means formulating predefined operating rules or AI models of desired characteristics by applying a learning process to a plurality of learning data. The learning may be performed in the device itself that executes the AI ​​according to the embodiment and / or may be implemented by a separate server / system.

[0044] The AI ​​model may consist of multiple neural network layers. Each layer has multiple weight values, and the layer operation is performed by the calculation of the previous layer and the operation of multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0045] A learning process is a method for training a predetermined target device (e.g., a robot) using a plurality of learning data to enable, allow, or control the target device to make a determination or prediction. Examples of learning processes include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0046] In an embodiment, a display (150) is configured to display a story including a first image, a second image, and a graphical representation generated between the first image and the second image to create continuity in the story. The display (150) is implemented using touch-sensitive technology and includes one of a liquid crystal display (LCD), a light emitting diode (LED), and the like.

[0047] although Figure 1 The hardware elements of the electronic device (100) are shown, but it should be understood that other embodiments are not limited thereto. In embodiments, the electronic device (100) may include fewer or greater numbers of elements. In addition, the labels or names of the elements are for illustrative purposes only and do not limit the scope of the present invention. One or more components may be combined together to perform the same or substantially similar functions.

[0048] Figure 2 is a flowchart ( 200 ) illustrating a method for creating continuity in a story through an electronic device ( 100 ) according to an embodiment disclosed herein.

[0049] Reference Figure 2 In step 202, the method includes the electronic device (100) receiving a first image and a second image as input. Figure 1 In the illustrated electronic device ( 100 ), the story management controller ( 140 ) is configured to receive a first image and a second image as input.

[0050] At step 204, the method includes the electronic device (100) determining a plurality of parameters associated with the first image. The plurality of parameters associated with the first image include scene elements in the first image, actions of the scene elements in the first image, and a theme formed by the scene elements in the first image. For example, in Figure 1 In the illustrated electronic device (100), the story management controller (140) is configured to determine a plurality of parameters associated with the first image.

[0051] At step 206, the method includes the electronic device (100) determining a plurality of parameters associated with the second image. The plurality of parameters associated with the second image include scene elements in the second image, actions of the scene elements in the second image, and a theme formed by the scene elements in the second image. For example, in Figure 1 In the illustrated electronic device (100), the story management controller (140) is configured to determine a plurality of parameters associated with the second image.

[0052] At step 208, the method includes the electronic device (100) generating a graphical representation to connect the first image with the second image based on a plurality of parameters associated with the first image and a plurality of parameters associated with the second image. Figure 1 In the electronic device (100) shown, the story management controller (140) is configured to generate a graphical representation to connect the first image with the second image based on a plurality of parameters associated with the first image and a plurality of parameters associated with the second image.

[0053] A graphical representation is generated to connect a first image with a second image by: (i) predicting a bridging event connecting the first image with the second image based on a plurality of parameters associated with the first image and a plurality of parameters associated with the second image; and (ii) generating a graphical representation of the bridging event to connect the first image with the second image.

[0054] (i) predicting a bridging event that connects a first image to a second image by creating a text summary for the first image and the second image based on a plurality of parameters associated with the first image and a plurality of parameters associated with the second image. Furthermore, generating a first image feature for the first image based on the text summary created for the first image, and generating a second image feature for the second image based on the text summary created for the second image, to predict a bridging event for connecting the first image to the second image.

[0055] (ii) generating a graphical representation of a bridging event to connect the first image with the second image by the following steps: comparing the first image and the second image based on a theme formed by scene elements in the first image and a theme formed by scene elements in the second image, determining that an image relationship distance between the first image and the second image is less than a first threshold, and generating a graphical representation of a bridging event to connect the first image with the second image when the image relationship distance between the first image and the second image is less than the first threshold.

[0056] At step 210, the method includes the electronic device (100) displaying a story including a first image, a second image, and a graphical representation generated between the first image and the second image. Figure 1 In the electronic device (100) shown, the story management controller (140) is configured to display a story including a first image, a second image, and a graphical representation generated between the first image and the second image.

[0057] The various actions, acts, blocks, steps, etc. in the method may be performed in the order presented, in a different order, or simultaneously. Furthermore, in some embodiments, some of the actions, acts, blocks, steps, etc. may be omitted, added, modified, skipped, etc. without departing from the scope of the invention.

[0058] Figure 3 is a block diagram of a system for creating continuity in a story according to embodiments disclosed herein.

[0059] Reference Figure 3 The system (3000) for creating continuity in a story includes a story management controller (140) and a metaverse story generator (340). The metaverse story generator (340) includes a metaverse scene creator (341), a meta-action animation generator (342), and a meta-story generator (343). The metaverse story generator (340) may be included in the electronic device (100) or implemented outside the electronic device (100).

[0060] In step 301, a plurality of images of a story are input into a story theme analyzer (142) of a story management controller (140). The story theme analyzer (142) analyzes themes formed by scene elements in the plurality of images of the story. The story theme analyzer (142) summarizes the entire story including the plurality of images into a detailed text summary based on the theme of the story. The detailed text summary is input into a scene analyzer (143).

[0061] At step 302, a scene analyzer (143) receives a detailed text summary of the entire story and predicts a scene connecting one image to another image based on parameters associated with the image. The scene analyzer (143) generates image features for each of the plurality of images based on the text summary of each image.

[0062] In step 303, a relational proximity detector (144) is used to detect a relational proximity distance between consecutive images including an object in a story. Here, the relational proximity detector (144) detects the degree of relevance of the images based on a theme formed by scene elements in a plurality of images of the story. The relational proximity distance may refer to "image relational distance", "visual scene distance" or "collaborative visual scene distance" in the claims of the present disclosure.

[0063] At step 304, the image sequencer (145) identifies features and similarities of objects present in the plurality of images. The image sequencer (145) arranges and groups images having similarity scores above a threshold in order based on the relationship proximity distance between the images and the features and similarities of the objects present in the images.

[0064] At step 305 , a bridging event predictor ( 146 ) is used to predict a bridging event for connecting consecutive images based on themes formed by scene elements in different images.

[0065] At step 306 , the text summarizer ( 147 ) receives the plurality of images as input and generates a text summary for each of the plurality of images that captures the most important elements, themes, scenes, etc. In addition, the text summarizer ( 147 ) creates an understanding of how the images fit into the story to generate a graphical representation of bridging events to connect sequential images of the story.

[0066] At step 307 , the metaverse scene creator ( 341 ) of the metaverse story generator ( 340 ) creates a metaverse scene based on the text summary of each of the plurality of images received from the text summarizer ( 147 ).

[0067] At step 308, the meta-action animation generator (342) generates a graphical representation of a bridging event to connect consecutive images of the story. The graphical representation of the bridging event includes, but is not limited to, a metaverse animation.

[0068] At step 309 , the meta-story generator ( 343 ) generates story continuity by inserting meta-universe animations between consecutive images.

[0069] At step 310 , a story having a first image, a meta-image, and a second image is displayed to enhance a user's story viewing experience in the metaverse.

[0070] Figure 4 is an example showing that images having gaps are bridged to achieve continuity according to the prior art.

[0071] Typically, a specific strategy is applied and a set of photos is selected for automatic story creation. Also, in case of manual story creation, a set of photos are grouped together to create a story. But in this case, there are gaps between consecutive photos in the story regarding actions, events, characters, scenes, etc., and there is no continuity from one photo to another. Figure 4 , consider three images from a story, image 1 (401), image 2 (402) and image 3 (403), there are gaps between image 1 (401) and image 2 (402) and between image 2 (402) and image 3 (403). Conventionally, there is no bridging of the gaps between images to make the images in the story have continuity. Therefore, there is a need to bridge the gaps between images to make the images in the story have continuity, which is addressed in the proposed method.

[0072] Figure 5a and Figure 5b is an example illustrating a process for bridging a gap between two images according to embodiments disclosed herein.

[0073] Reference Figure 5a and Figure 5b, receiving image 1 (510a, 510b) and image 2 (520a, 520b) as input. Determining parameters associated with image 1 (510a, 510b) and image 2 (520a, 520b) to predict a bridging event for connecting image 1 (510a, 510b) with image 2 (520a, 520b). Predicting a bridging event based on parameters associated with image 1 (510a, 510b) and image 2 (520a, 520b). Generating a graphical representation (530a, 530b) of the bridging event to connect image 1 (510a, 510b) with image 2 (520a, 520b) to bridge a gap between image 1 (510a, 510b) and image 2 (520a, 520b). The graphical representation (530a, 530b) of the bridging event includes, but is not limited to, a metaverse animation.

[0074] Figure 6 is an example showing story visualization based on scene distance according to embodiments disclosed herein.

[0075] Reference Figure 6 , consider a plurality of images (601-609) from a story. The distances between the plurality of images (601-609) in the story are different for different pairs of images. For example, the distance between image 4 (604) and image 5 (605) is smaller when compared to the distance between image 7 (607) and image 8 (608). Therefore, the distances between consecutive images of the story must be determined to generate a bridging event for connecting the consecutive images.

[0076] Multiple images (601-609) with higher levels of similarity are grouped into small sets. These sets are called spans in a story. A story contains multiple spans. In order to predict and generate metaverse animations, it is necessary to understand the spans of a story. Based on the similarity scores between consecutive images (such as, for example, image 4 (604) and image 5 (605) in the multiple images (601-609), a visual scene distance and / or a collaborative visual scene distance between multiple spans of the multiple images (601-609) is determined. The collaborative visual scene distance may refer to the visual scene distance. When the visual scene distance and / or the collaborative visual scene distance between the multiple spans of the multiple images (601 to 609) is less than a second threshold, a graphical representation of a bridging event is generated to connect the multiple spans of the multiple images (601 to 609) in the story.

[0077] In an embodiment, the electronic device (100) generates a plurality of spans of the plurality of images based on similarity scores between images in the plurality of images. The electronic device (100) determines trajectories of objects available in the plurality of images. The electronic device (100) identifies features of the objects available in the plurality of images based on the determined trajectories of the objects available in the plurality of images. The electronic device (100) determines whether the similarity scores between the plurality of images are above a third threshold based on the identified features of the objects available in the plurality of images. The electronic device (100) generates a plurality of spans of the plurality of images based on the similarity scores between the plurality of images.

[0078] Figure 7 is a block diagram illustrating a multi-image story summary according to embodiments disclosed herein.

[0079] In an embodiment, the story theme analyzer (142) is trained by taking multiple images as input and learning their properties and relationships. The story theme analyzer (142) takes multiple images in a story as input and generates a text summary of the story that captures the most important elements, themes, scenes, etc. In an embodiment, the above training and reasoning operations can be completed in the text summarizer (147).

[0080] Reference Figure 7 , at step 710 , image 1 , image 2 , image 3 , and image 4 are selected from the story, and each image is summarized as a text summary.

[0081] At step 720 , the textual summaries of each image are combined and summarized into an aggregate summary of Image 1 , Image 2 , Image 3 , and Image 4 .

[0082] At step 730 , the encoder is configured to encode and determine the most important elements, themes, scenes, etc. based on images 1 - 4 and the aggregate summary of images 1 , 2 , 3 , and 4 .

[0083] At step 740, the decoder is configured to convert the encoded digital stream into a text summary of the story's Image 1 - Image 4 based on the most important elements of the image, theme, scene, etc.

[0084] Figure 8 is a block diagram illustrating a process of creating a story-based text summary for a single image according to embodiments disclosed herein.

[0085] In an embodiment, a text summarizer (147) is trained based on the story themes formed by the scene elements of each image. The text summarizer (147) takes as input the text summaries of the image and the story and learns their properties and relationships. The text summarizer (147) takes as input the text summaries of the image and the story and generates a text summary of the image that captures the most important elements, themes, scenes, etc. In addition, the text summarizer (147) captures how the image fits into the story.

[0086] Reference Figure 8 , in step 810, the text summarizer (147) receives the text summaries of the images and stories output in step 740 as input, and performs separate summaries based on elements, themes, scenes, etc.

[0087] At step 820 , the individual summaries of the images and the textual summary of the stories output at step 740 are combined to output a summary.

[0088] At step 830, the individual summaries of the images and the textual summaries of the stories and the aggregated summaries are encoded to determine parameters associated with the images.

[0089] At step 840, the decoder is configured to decode the encoded digital stream and output a text summary of each image based on parameters associated with the image.

[0090] The embodiments of the present disclosure may also be implemented as a storage medium including instructions that can be executed by a computer (such as a program module executed by a computer). Computer-readable media can be any available media that can be accessed by a computer and includes all volatile / non-volatile media and removable / non-removable media.

[0091] In addition, computer-readable media may include all computer storage media and communication media. Computer storage media include all volatile / non-volatile media and removable / non-removable media implemented by a specific method or technology for storing information (such as computer-readable instruction codes, data structures, program modules or other data). Communication media may generally include computer-readable instructions, data structures or other data in modulated data signals, such as program modules. In addition, computer-readable storage media may be provided in the form of non-transitory storage media.

[0092] A "non-transitory storage medium" is a tangible device and simply means that it does not contain signals (e.g., electromagnetic waves). The term does not distinguish between a case where data is semi-permanently stored in a storage medium and a case where data is temporarily stored in a storage medium. For example, a non-transitory recording medium may include a buffer that temporarily stores data.

[0093] According to an embodiment of the present disclosure, the method according to various disclosed embodiments may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product is distributed in the form of a device-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be distributed (e.g., downloaded or uploaded) directly and online through an application store or between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be at least temporarily stored in a device-readable storage medium (such as a memory of a manufacturer's server, a memory of an application store's server, or a memory of a relay server), or may be temporarily generated.

[0094] The foregoing description of specific embodiments will fully reveal the general nature of the embodiments herein so that others can easily modify and or adapt various applications of such specific embodiments by applying current knowledge without departing from the general concepts, and therefore, such adaptations and modifications should and are intended to be understood as being within the meaning and scope of equivalent forms of the disclosed embodiments. It should be understood that the wording or terminology used herein is for descriptive and not limiting purposes. Therefore, although the embodiments herein have been described according to preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced by modification within the scope of the embodiments as described herein.

Claims

1. A method for creating continuity in a story, the method comprising: receiving as input at least one first image and at least one second image; determining a plurality of parameters associated with the at least one first image; determining a plurality of parameters associated with the at least one second image; generating a graphical representation to connect the at least one first image with the at least one second image based on the plurality of parameters associated with the at least one first image and the plurality of parameters associated with the at least one second image; as well as A story including the at least one first image, the at least one second image, and a graphical representation generated between the at least one first image and the at least one second image is displayed.

2. The method according to claim 1, in, The plurality of parameters associated with the at least one first image include scene elements in the at least one first image, actions of the scene elements in the at least one first image, and a theme formed by the scene elements in the at least one first image, and The multiple parameters associated with the at least one second image include scene elements in the at least one second image, actions of the scene elements in the at least one second image, and a theme formed by the scene elements in the at least one second image.

3. The method according to any one of claims 1 to 2, wherein: The step of generating the graphical representation to connect the at least one first image with the at least one second image comprises: predicting at least one bridging event connecting the at least one first image with the at least one second image based on the plurality of parameters associated with the at least one first image and the plurality of parameters associated with the at least one second image; and The graphical representation of the bridging event is generated to connect the at least one first image with the at least one second image.

4. The method of claim 3, wherein: The step of predicting the at least one bridging event connecting the at least one first image with the at least one second image comprises: creating a text summary for the at least one first image and the at least one second image based on the plurality of parameters associated with the at least one first image and the plurality of parameters associated with the at least one second image; generating at least one first image feature for the at least one first image based on the text summary created for the at least one first image; generating at least one second image feature for the at least one second image based on the text summary created for the at least one second image; and At least one bridging event for connecting the at least one first image and the at least one second image is predicted based on the at least one first image feature generated for the at least one first image and the at least one second image feature generated for the at least one second image.

5. The method according to any one of claims 3 to 4, wherein: The step of generating the graphical representation of the bridging event to connect the at least one first image with the at least one second image comprises: comparing the at least one first image and the at least one second image based on a subject formed by the scene elements in the at least one first image and a subject formed by the scene elements in the at least one second image; determining whether an image relationship distance between the at least one first image and the at least one second image is less than a first threshold based on a comparison between the at least one first image and the at least one second image; and In a case where the image relationship distance between the at least one first image and the at least one second image is less than the first threshold, the graphical representation of the bridging event is generated to connect the at least one first image with the at least one second image.

6. The method according to any one of claims 1 to 5, further comprising: Acquire multiple images; identifying features of an object available in each of the plurality of images; determining a similarity score for grouping images of the plurality of images having similar features of the object into spans; generating a plurality of spans of the plurality of images based on the similarity scores between images in the plurality of images; determining a visual scene distance between the plurality of spans of the plurality of images; determining whether the visual scene distance between the plurality of spans of the plurality of images is less than a second threshold; as well as In a condition where the visual scene distance between the plurality of spans of the plurality of images is less than the second threshold, the graphical representation of the bridging event is generated to connect the plurality of spans of the plurality of images.

7. The method of claim 6, wherein: The step of generating a plurality of spans of the plurality of images based on the similarity scores between images in the plurality of images comprises: determining a trajectory of the object available in the at least one first image of the plurality of images and a trajectory of the object available in the at least one second image of the plurality of images; identifying features of the object available in the at least one first image and features of the object available in the at least one second image in the plurality of images based on the determined trajectory of the object available in the at least one first image in the plurality of images and the at least one second image; determining whether the similarity score between the at least one first image among the plurality of images and the at least one second image among the plurality of images is above a third threshold based on the identified features of the object available in the at least one first image among the plurality of images and the at least one second image; and A plurality of spans of the plurality of images are generated based on the similarity scores between the at least one first image of the plurality of images and the at least one second image of the plurality of images.

8. An electronic device (100) for creating continuity in a story, the electronic device (100) comprising: Memory (110); A processor (120), coupled to the memory (110); a communicator (130) coupled to the memory (110) and the processor (120); as well as A story management controller (140) is coupled to the memory (110), the processor (120) and the communicator (130), and is configured to: receiving as input at least one first image and at least one second image, determining a plurality of parameters associated with the at least one first image, determining a plurality of parameters associated with the at least one second image, generating a graphical representation to connect the at least one first image with the at least one second image based on the plurality of parameters associated with the at least one first image and the plurality of parameters associated with the at least one second image, and A story including the at least one first image, the at least one second image, and a graphical representation generated between the at least one first image and the at least one second image is displayed.

9. The electronic device (100) as claimed in claim 8, in, The plurality of parameters associated with the at least one first image include scene elements in the at least one first image, actions of the scene elements in the at least one first image, and a theme formed by the scene elements in the at least one first image, and The multiple parameters associated with the at least one second image include scene elements in the at least one second image, actions of the scene elements in the at least one second image, and a theme formed by the scene elements in the at least one second image.

10. The electronic device (100) according to any one of claims 8 to 9, wherein: The story management controller (130) is also configured to: predicting at least one bridging event connecting the at least one first image with the at least one second image based on the plurality of parameters associated with the at least one first image and the plurality of parameters associated with the at least one second image, and The graphical representation of the bridging event is generated to connect the at least one first image with the at least one second image.

11. The electronic device (100) according to claim 10, wherein: The story management controller (130) is also configured to: creating a text summary for the at least one first image and the at least one second image based on the plurality of parameters associated with the at least one first image and the plurality of parameters associated with the at least one second image; generating at least one first image feature for the at least one first image based on the text summary created for the at least one first image; generating at least one second image feature for the at least one second image based on the text summary created for the at least one second image; as well as At least one bridging event for connecting the at least one first image and the at least one second image is predicted based on the at least one first image feature generated for the at least one first image and the at least one second image feature generated for the at least one second image.

12. The electronic device (100) according to any one of claims 10 to 11, wherein: The story management controller (140) is also configured to: comparing the at least one first image and the at least one second image based on a subject formed by the scene elements in the at least one first image and a subject formed by the scene elements in the at least one second image; determining whether an image relationship distance between the at least one first image and the at least one second image is less than a first threshold based on a comparison between the at least one first image and the at least one second image; as well as In a case where the image relationship distance between the at least one first image and the at least one second image is less than the first threshold, the graphical representation of the bridging event is generated to connect the at least one first image with the at least one second image.

13. The electronic device (100) according to any one of claims 8 to 12, wherein: The story management controller (140) is also configured to: Acquire multiple images; identifying features of an object available in each of the plurality of images; determining a similarity score for grouping images of the plurality of images having similar features of the object into spans; generating a plurality of spans of the plurality of images based on the similarity scores between images in the plurality of images; determining a visual scene distance between the plurality of spans of the plurality of images; determining whether the visual scene distance between the plurality of spans of the plurality of images is less than a second threshold; as well as In a condition where the visual scene distance between the plurality of spans of the plurality of images is less than the second threshold, the graphical representation of the bridging event is generated to connect the plurality of spans of the plurality of images.

14. The electronic device (100) according to claim 13, wherein: The story management controller (140) is also configured to: determining a trajectory of the object available in the at least one first image of the plurality of images and a trajectory of the object available in the at least one second image of the plurality of images; identifying features of the object available in the at least one first image and features of the object available in the at least one second image in the plurality of images based on the determined trajectory of the object available in the at least one first image in the plurality of images and the at least one second image; determining that the similarity score between the at least one first image in the plurality of images and the at least one second image in the plurality of images is above a third threshold based on the identified features of the object available in the at least one first image in the plurality of images and the at least one second image; as well as A plurality of spans of the plurality of images are generated based on the similarity scores between the at least one first image of the plurality of images and the at least one second image of the plurality of images.

15. A computer-readable medium comprising instructions, which when executed cause at least one processor of an electronic device to perform operations corresponding to the method of any one of claims 1 to 7.

Images