System and method for associating text summaries with content media

Abstract

System and method for associating text summaries with content media. A method and system for associating text summaries with data representing media content is provided. The method may include the following steps: receiving a plurality of text summaries, each text summary representing an event; pairing each received text summary with each of a plurality of data representing media content, respectively, through a neural network to generate a plurality of text-data pairs; and associating a first selected text summary with a first data based on similarities between content features extracted from each received text summary and content features extracted from each of the data in each of the plurality of text-data pairs.

Description

Technical Field

[0001] The present disclosure relates to content media book generation, and in particular, to systems and methods for associating text summaries with content media to generate a content media book. Background Art

[0002] With the prevalence of media capture devices (e.g., cameras in mobile devices, image capture devices, video capture devices, etc.), users often encounter situations where they acquire a large or overwhelming amount of digital content media (such as photos and videos) during the course of attending events, going on vacation, or throughout a lifetime. Users may wish to edit the collection into a smaller collection of photos, videos, or a combination thereof that is more coherent, shows the best quality media, and tells a story. Such collections are often shared via "books" or "collections" on social photo sharing platforms, or compiled into books (referred to herein as "photo books") for printing and physical distribution. However, with related art systems, the creation of such visual summaries must often be laboriously performed by users editing and sorting their photos, videos, etc. with the help of desktop or online book or book creation tools. Several related art systems can provide automation that assists users in selecting the best shots, grouping them based on content features or themes or by people featured in the content media, and otherwise enhancing their specific levels of automation with captions and other visual effects.

[0003] However, existing related art media organization tools work by starting with obtaining a user-provided photo collection and proceeding to apply analysis to determine the content and quality of the photos. The related art system can then select groups of photos to represent larger collections and often organize them into presentations of multiple photos per page, and allow users to reorder and otherwise reorganize the photos on the page. Existing related art systems do not address the reverse situation, where a user can provide captions or text summaries for portions of a photo book or album, thereby providing some guidance on how or what a story can be told. Example embodiments of the present application can address this shortcoming. Summary of the invention

[0004] Aspects of the present application may relate to a method for associating a text summary with data representing media content. The method may include the following steps: receiving a plurality of text summaries, each text summary representing an event; pairing each received text summary with each of a plurality of data representing media content, respectively, through a neural network to generate a plurality of text-data pairs; and associating a first selected text summary with a first data based on similarities between content features extracted from each received text summary and content features extracted from each data in each of the plurality of text-data pairs.

[0005] Another aspect of the present application may relate to a non-volatile computer-readable medium having a program stored therein, the program being used to cause a computer to execute a method of associating a text summary with data representing media content. The method may include the following steps: receiving a plurality of text summaries, each text summary representing an event; pairing each received text summary with each of a plurality of data representing media content, respectively, through a neural network to generate a plurality of text-data pairs; and associating a first selected text summary with a first data selected from a plurality of data based on the similarity between content features extracted from each received text summary and content features extracted from each data in each of the plurality of text-data pairs.

[0006] Another aspect of the present application relates to a computer device configured to associate a text summary with data representing media content. The computer device may include: a memory storing a plurality of data representing media content; and a processor performing a process of visualizing the recommended media content based on the observed media content. The process may include: providing a user with an interface for submitting a plurality of text summaries, each text summary representing an event; pairing each provided text summary with each of a plurality of data stored in the memory through a neural network to generate a plurality of text-data pairs; and associating a first selected text summary with a first data selected from a plurality of data based on the similarity between content features extracted from each received text summary and content features extracted from each of the plurality of text-data pairs in the plurality of text-data pairs.

[0007] Some other aspects of the present application relate to a computer device configured to associate a text summary with data representing media content. The computer device may include: a device for receiving a plurality of text summaries, each of which represents an event; a device for pairing each of the received text summaries with each of the plurality of data representing the media content, respectively, to generate a plurality of text-data pairs; and a device for associating a selected first text summary with a first data selected from the plurality of data based on similarities between content features extracted from each of the received text summaries and content features extracted from each of the plurality of text-data pairs. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The patent or application file contains at least one drawing executed in color. Copies of the patent or patent application publication with color drawings will be provided by the Office upon request and payment of the necessary fee.

[0009] Figure 1 A flow chart of a process for generating a photo book according to an example embodiment of the present application is illustrated.

[0010] Figure 2 A flowchart of a process for generating a photo book according to another example embodiment of the present application is illustrated.

[0011] Figure 3 A flowchart of a sub-process for forming associated pairs based on content similarity according to an example embodiment of the present application is illustrated.

[0012] Figure 4 A flowchart of a sub-process for forming associated pairs based on content similarity according to another exemplary embodiment of the present application is illustrated.

[0013] Figure 5 A flowchart of a process for generating a photo book according to another example embodiment of the present application is illustrated.

[0014] Figure 6 and Figure 7 A user interface (UI) that may be used to control a system according to an example embodiment of the present application is illustrated.

[0015] Figure 8 An example computing environment having an example computer device suitable for use in some example embodiments of the present application is illustrated. DETAILED DESCRIPTION

[0016] The following specific embodiments provide additional details of the drawings and example embodiments of the present application. For the sake of clarity, the reference numerals and descriptions of redundant elements between the drawings are omitted. The terms used throughout the specification are provided as examples and are not intended to be limiting. For example, the use of the wording "automatic" depends on the desired embodiment of a person of ordinary skill in the art who practices the embodiments of the present application and can relate to fully automatic or semi-automatic embodiments, which relate to user or operator control on the particular aspects of the embodiments. Further, the order words such as "first", "second", "third", etc. are simply used for marking purposes in the specification and claims, and should not be limited to refer to the actions or projects that occur in the described order. Without departing from the scope of the present application, actions or projects can be sorted into different orders, or can be executed in parallel or dynamically.

[0017] In this application, the terms "image", "content media", "content media representation" or "media representation" may be used interchangeably to describe one or more of a photograph, video, three-dimensional or 360° image recording, drawing, painting, sketch, computer-generated image, or any other visual representation that may be produced, captured, or otherwise represent an event or occurrence in the real world. These terms may also include recordings or other audio representations that may be produced, captured, or otherwise represent an event or occurrence in the real world. An "image", "content media", "content media representation" or "media representation" may be captured by any media capture device, including but not limited to: a camera, a digital recorder, an analog recorder, a mobile communication device, or any other media capture device that is obvious to a person skilled in the art. A "content media", "content media representation" or "media representation" may be stored as a data file or other data structure on a computer-readable medium, including but not limited to: a magnetic storage device, an optical storage device, a solid-state storage device, an organic storage device, or any other storage device that is obvious to a person skilled in the art. Further, the computer-readable medium may include a local storage device, a cloud-based storage device, a remotely located server, or any other storage device that may be apparent to one of ordinary skill in the art.

[0018] Further, in this application, the terms "caption" or "text summary" may all be used interchangeably to refer to a descriptive text-based summary that may represent the content of one or more of the "image", "content media", "content media representation" or "media representation".

[0019] According to an example embodiment of the present application, a user may provide a series of captions or text summaries related to an event, and the system may then provide a matching set of photos, videos, etc. from the user's collection to "exemplify" the captions. This may empower users in situations where they have memories of what situations and events they captured but have limited tools and resources to appropriately retrieve these moments from the collection they have at hand.

[0020] For example, a user returns from vacation with a memory card full of photos. Using a system according to an example embodiment, the user can then provide an overview of the moments they want to summarize into a photo book or album by providing a series of captions describing the events they want to depict, such as "We had a great day at the beach," "Dad tried parasailing," or "The tacos were delicious." The system can then rank all images according to their relevance to each caption and provide images of beach time, parasailing, and tacos, respectively, as corresponding to the captions.

[0021] In another example embodiment, a user may post to a social media account or microblogging platform (e.g., TWITTER, etc.) during travel, and the user may provide the system with access to the user's social media account or microblog, and the posts may be retrieved by the system to be used as captions or text summaries to be illustrated with photographs or other media content to create a photo book or album.

[0022] In other example embodiments, the system may be used to index and identify relevant portions of a large content media representation, such as a surveillance video, a point-of-view or egocentric video (e.g., a GoPro video), a recording of an educational lecture, a recording of an extended interview, or any other type of large content media representation that may be apparent to one of ordinary skill in the art. For example, the system may be used to identify one or more image frames or audio portions in a large content media representation that are associated with a content feature based on a caption or text summary provided by a user.

[0023] Further, other example embodiments may include a system configured to generate a photo book from user-provided captions by querying a collection of images. In these embodiments, the input to the system may be a user's photo collection or a temporal, thematic, or geographic subsection of their collection, and a series of captions or other textual summaries that the user wants to "exemplify" using the provided collection. The system may work by learning from (e.g., training with) artificially created photo books or photo collections (e.g., training data) that are provided from multiple users of a book or photo collection creation tool. In these embodiments, captions (of the training data) that the user has selected to be associated with a subset of the photos (of the training data) within the book or book may be extracted, as well as additional photos from the user's collection and their status as included (in the training data) with or without the captions. Using this training data set, the system may learn representations of the visual content of the provided captions or text summaries and the images and associated metadata. The system may then use a model that predicts the relevance of each image to the associated captions. These representations and models can then be used to predict whether a user will select a given image to be associated with a given caption. Using these predictions, images can be ranked according to their predicted relevance for each caption, and then an image can be selected for each caption using a round-robin selection scheme.

[0024] Various example embodiments discussed herein demonstrate a method for instantiating captions using a machine learning matching system, wherein the relevance of images to captions can be learned based on historical examples of caption-image pairs provided by users. The method can involve multiple components that provide automatic associations between images and captions:

[0025] - large databases of images that have been manually sub-selected into groups with associated captions or text summaries;

[0026] - a method for representing the semantic content of both images and associated captions in a database; and

[0027] -Methods for predicting the relevance of a given image to a provided caption.

[0028] Further aspects of the example embodiments are discussed in detail below with reference to the accompanying drawings.

[0029] Figure 1 1 illustrates a flow chart of a process 100 for generating a photo book according to an example embodiment of the present application. The process 100 may be performed by a computing device in a computing environment, such as the one discussed below. Figure 8 An example computer device 805 of the illustrated example computing environment 800. Although the elements of the process 100 may be illustrated in a particular order, the example embodiments are not limited to the particular order illustrated. Without departing from the scope of the present application, the example embodiments may include actions that are sequenced into a different order as may be apparent to one of ordinary skill in the art or actions that may be performed in parallel or dynamically.

[0030] like Figure 1 As an example, user input is received by the system at 105. The user input may include a user's image set and a user-provided caption that the user wants to illustrate. After receiving the user input, the system may generate a caption-image pair at 110 by pairing each provided caption with each possible image in the user-provided image set.

[0031] At 115, features may be extracted for both the caption and the image to produce a characterized embedding associated with both the caption and the image. The feature extraction at 115 may be performed using a neural network trained using training data at 135. The training data may include a collection of images or photo books that have been manually generated by a user of a photo album editing or photo book generation platform that has been captioned, which may be website-based, mobile application-based, or installed on a local machine. For each image / caption pair, the features (embeddings) may be concatenated together.

[0032] In some example embodiments, content feature extraction may include generating a representation space for user-provided captions and candidate images that may be applied to exemplify these captions. To represent the content features of each candidate photo and each caption, subnetworks may be trained on different tasks to learn embeddings.

[0033] For example, text embeddings can be learned using existing techniques (e.g., "word2vec") to learn vector encodings that represent words extracted from a caption or text summary. Further, in some example embodiments, text recognition techniques (e.g., optical character recognition, etc.) can also be used to extract words from images. This type of network can have a simple architecture that takes a large-scale single-hotspot representation of the vocabulary and then maps it with the help of a very low-dimensional set of nodes to predict terms in the same single-hotspot representation in the final layer. The network can be trained by providing a large corpus of text and using a single-hotspot representation of the current term and training the network to predict subsequent terms in the text. For a user-provided text caption, each term can be passed through the network and the representation of each term can be maintained in an intermediate low-dimensional layer. The representation values ​​can be used as feature representations of the text terms.

[0034] Further, multiple methods can be used to embed sentences or phrases. For example, neural methods that use the context of neighboring words or sentences can be effectively used by taking the word2vec representation discussed as input and embedding the caption into a low-dimensional embedding.

[0035] To extract features that represent images, a deep convolutional neural network can be trained on a large corpus (e.g., "ImageNet"). In this type of network, images can be passed through a series of convolutional layers with maximum pooling and ReLU activation. After the convolutional layers, multiple flattened fully connected layers are applied for the final task of classifying the image, typically into one of the ImageNet classes. Once the network is trained on this data, a user-provided image can be passed through the network, and the penultimate layer output can be used as a feature representation applied toward the selection task discussed below.

[0036] In some example embodiments, the image representation may be further enhanced with metadata reflecting additional information cues related to the capture of the image, such as timestamps, geographic coordinates, and camera settings. Further, in some example embodiments, an automated system may be employed to enhance such metadata with information extracted from the image, such as the identity of the people in the image or the projected aesthetic quality of the image. Additionally, data indicating past preferences of a particular user or past album or photo book generation choices may be fed into the model to further improve feature extraction and image selection.

[0037] At 120, each image / caption pair may be passed through a scoring model developed by a neural network based on the training data to predict the relevance of each image to each caption. For each caption, a score model may then be used to rank the image. In some example embodiments, once a content representation is obtained from 115, a model that learns the likelihood of an image from a larger set of users in relation to the captions provided may be applied to select, score and rank caption / image pairs at 120. This may be done by compiling sets of images belonging to various users and their subsequent albums or photo books generated from these photo collections along with the captions provided by various users for each image or group of images. For each caption that a user manually uses in a photo book, a training pair may be created for the caption and each image in the user's entire image collection. A binary classifier may then be used to model whether the user actually selected the photo as included with the caption. The caption may be passed through a neural network to extract an embedding for modeling the text. Similarly, each image is passed through an image modeling subnetwork, and the embedding is used there to represent the image. Metadata features may also be extracted. For each image-caption pair, these representations can be concatenated together as input for further modeling. A deep network of dense layers can then be applied, in which the final layer is a softmax transfer function layer that predicts inclusion ("1") or exclusion ("0") from the caption set. During ranking, each user-provided caption is paired with each image in his or her collection and scored based on the relevance of the image to the caption. Images are then ranked based on the scores returned.

[0038] Based on the ranking, images can then be selected for each caption at 125 without replacement (e.g., without providing new images) until a desired number of images are found for each caption. For example, images in the top k sets or images above a certain threshold can be returned as the set of matching photos for a given caption. In some example embodiments, a greedy selection algorithm can be used at 125 to select the best (e.g., highest ranked) image for each caption. Further, in some example embodiments, once an image is selected to be associated with a given caption, the same image will not be associated with any other caption in a given album or photo book. Example embodiments of this are discussed in more detail below.

[0039] Once the desired number of images are associated with each user-provided caption, an image album or photo book may be generated according to the user settings at 130, and the process may end. In some example embodiments, the user settings may be provided by a user interface (such as discussed in more detail below). Figure 6 and Figure 7In some example embodiments, the user settings may be provided by settings associated with the image capture device and / or additional training data based on the user's previous albums or photo books.

[0040] Figure 2 A flowchart of a process 200 for generating a media content book according to another example embodiment of the present application is illustrated. Some aspects of the process 200 may be similar to those described above with reference to Figure 1 The process 200 discussed above is similar to the process 100 discussed above, and therefore, redundant discussion may be omitted here. The process 200 may be performed by a computing device in a computing environment, such as the one discussed below. Figure 8 An example computer device 805 of the illustrated example computing environment 800. Although the elements of process 200 may be illustrated in a particular order, the example embodiments are not limited to the particular order illustrated. Without departing from the scope of the present application, the example embodiments may include actions that are sequenced into a different order as may be apparent to one of ordinary skill in the art or actions that may be performed in parallel or dynamically.

[0041] Again, user input is received by the system at 205. In process 200, the user input may include a user-provided text summary or caption that the user wants to illustrate. The image or content representation may have been previously provided to the system, or the system may have been integrated into an image management system or social media platform. In some example embodiments, the text summary may be provided by extracting a social media post or microblog post from a social media platform.

[0042] After receiving user input, the system can generate text summary-content representation pairs at 210 by pairing each provided text summary with each possible content representation.

[0043] At 215, a content similarity value is determined for each text summary-content representation pair. Figure 1 As discussed in the example embodiments of , this can include extracting content features of both the text summary and the content representation image to produce feature embeddings associated with both the caption and the image. Feature extraction can be performed using a neural network that has been trained using training data. The training data can include sets of images or photo books that have been manually generated by users of a captioned book editing or photo book generation platform that can be accessed via a website, mobile application, or software installed on a local machine. For each content representation / text summary pair, the extracted features (embedded objects) can be concatenated together.

[0044] Again, in order to represent the content features of each content representation and text summary, subnetworks can be trained on different tasks to learn embeddings. For example, text embeddings can include using existing techniques (e.g., "word2vec") to learn vector encodings representing words extracted from captions or text summaries. In some embodiments, as part of text extraction, text recognition techniques (e.g., optical character recognition, etc.) can also be used to extract words from images. The neural network can have a simple architecture that takes a large-size single-hotspot representation of the vocabulary and then maps it with the help of a very low-size set of nodes to predict the terms in the same single-hotspot representation in the final layer. This type of network can be trained by providing a large corpus of text and using a single-hotspot representation of the current term and training the network to predict subsequent terms in the text. For a user-provided text summary, each term can be passed through the network, and the representation of each term can be maintained in an intermediate low-size layer. The representation value can be used as a feature representation of the text term.

[0045] Further, multiple methods can be used to embed sentences or phrases. For example, neural methods that use the context of neighboring words or sentences can be effectively used by taking the word2vec representation discussed as input and embedding the caption into a low-dimensional embedding.

[0046] With respect to extracting content features that represent images, a deep convolutional neural network can be trained on a large corpus (e.g., "ImageNet"). In this network, images can be passed through a series of convolutional layers with maximum pooling and ReLU activation. After the convolutional layers, multiple flattened fully connected layers are applied for the final task of classifying the image, typically into one of the ImageNet classes. Once the network is trained on this data, a user-provided image can be passed through the network, and the penultimate layer output can be used as a feature representation applied toward the selection task discussed below.

[0047] In some example embodiments, the image representation may be further enhanced with metadata reflecting additional information about the capture of the image, such as timestamps, geographic coordinates, and camera settings, and the image representation may also use an automated system to enhance the metadata with information extracted from the image, such as the identity of the person in the image or the projected aesthetic quality of the image. Additionally, data indicating a particular user's past preferences or more past album or photo book generation selections may also be fed into the model to further improve feature extraction and image selection.

[0048] Once the content similarity is determined for each text summary-content representation pair, the first text summary can be associated with the first content representation based on the content similarity of each text summary-content representation pair at 220 to produce a first associated pair. Similarly, the second text summary can be associated with the second content representation based on the content similarity of each text summary-content representation pair at 225 to produce a second associated pair. Example implementations of 220 and 225 can be implemented using the following respectively: Figure 3 and Figure 4 In addition, in some example embodiments, 220 and 225 may be repeated based on the content similarity determined for each text summary-content representation pair until a desired number of associated pairs (e.g., three, four, five, etc. associated pairs of text summary-content representation) are obtained.

[0049] Once the desired number of associated pairs is generated, an image album or photo book may be generated according to the user settings at 230, and the process may end. In some example embodiments, the user settings may be provided by a user interface (such as discussed in more detail below). Figure 6 and Figure 7 In some example embodiments, the user settings may be provided by settings associated with the image capture device and / or additional training data based on the user's previous albums or photo books.

[0050] Figure 3 A flow chart of a subprocess 300 for forming association pairs based on content similarity is illustrated. The subprocess 300 may be included as part of a larger process for generating a content media album (such as discussed throughout this document). Figure 1 , Figure 2 , Figure 5 Subprocess 300 may be performed by a computing device in a computing environment, such as the following discussed. Figure 8 The example computer device 805 of the example computing environment 800 is shown. Figure 3 As used, the term "pair" may be a caption-image pair, a text summary-content representation pair, a text summary-data cluster pair, or any other text summary image representation pair that may be apparent to one of ordinary skill in the art.

[0051] Although the elements of subprocess 300 may be illustrated in a particular order, the example embodiments are not limited to the particular order illustrated. Without departing from the scope of the present application, the example embodiments may include actions that are sequenced into a different order as may be apparent to one of ordinary skill in the art or actions that may be performed in parallel or dynamically.

[0052] When creating a photo book or album, a user will sometimes attempt to "instantiate" multiple captions and compile all caption-image sets into a larger set. This situation may impose an additional constraint on the system in that, typically, each photo should only be associated with a single caption to avoid duplication of photos between larger books. In some exemplary embodiments, a greedy approach may be applied to match images to captions as illustrated by subprocess 300. In subprocess 300, pairs with the highest similarity retention may be iteratively selected for a photo book or image album until each caption has collected its required number of images or the number of "good" matches (matches above a certain threshold score) is exhausted for each caption-image set.

[0053] In subprocess 300, the set of ranked pairs is evaluated at 305 to determine whether the pair with the highest content similarity has been selected. If the pair with the highest content similarity has not been selected (No at 305), the pair with the highest content similarity is selected at 310 as the first associated pair, two components of the selected pair (e.g., the caption / text summary and the image / content representation / data cluster) are removed from further consideration, and subprocess 300 returns to 305.

[0054] Conversely, if the pair with the highest content similarity has already been selected (yes at 305), then at 315 the pair with the highest content similarity is selected from the remaining pairs as a second or subsequently associated pair, two components of the selected pair (e.g., the caption / text summary and the image / content representation / data cluster) are removed from further consideration, and subprocessing 300 proceeds to 320.

[0055] At 320, it is determined whether the number of association pairs selected so far is less than a threshold. In some example embodiments, the threshold may represent a user setting of the number of captions / text summaries or images / content representations / data clusters to be generated in the generated media content book. In some example embodiments, the threshold may be set using a user interface (such as discussed in more detail below). Figure 6 and Figure 7 If the number of associated pairs selected so far is less than the threshold value (yes at 320), the sub-process 300 returns to 305 and continues from there. Conversely, if the number of associated pairs selected so far is not less than the threshold value (no at 320), the sub-process 300 ends and the main process for generating a content media book can continue.

[0056] Figure 4 4. A flowchart of a subprocess 400 for forming association pairs based on content similarity is illustrated. Subprocess 300 may be included as part of a larger process for generating a content media album (such as discussed throughout this document). Figure 1 , Figure 2 , Figure 5 Subprocess 400 may be performed by a computing device in a computing environment, such as the following discussed. Figure 8 The example computer device 805 of the example computing environment 800 is shown. Figure 4 As used, the term "pair" may be a caption-image pair, a text summary-content representation pair, a text summary-data cluster pair, or any other text summary image representation pair that may be apparent to one of ordinary skill in the art.

[0057] Although the elements of subprocess 400 may be illustrated in a particular order, the example embodiments are not limited to the particular order illustrated. Without departing from the scope of the present application, the example embodiments may include actions that are sequenced into a different order as may be apparent to one of ordinary skill in the art or actions that may be performed in parallel or dynamically.

[0058] When creating a photo book or album, a user will sometimes attempt to "instantiate" multiple captions and compile all caption-image sets into a larger set. An additional constraint that this situation may impose on the system is that, typically, each photo should only be associated with a single caption to avoid duplication of photos between larger books. In some example embodiments, a round-robin image selection method is used in which a first caption selects its top-ranked image to include in its set, thereby removing that image from the candidates for all other caption sets. Then, a second caption takes its top-ranked image from the remaining images. This process can be repeated until all image sets select the desired number of images from the pool. Figure 4 Subprocess 400 in exemplifies an example of this method.

[0059] In subprocess 400, a first of the text summaries or captions is selected at 405. The selection of the first text summaries or captions may be random, may be based on an order of user entry, may be selected in response to user input, or may be selected based on previous user preferences.

[0060] Once a first text summary or caption is selected, the pair containing the selected first text summary and having the highest content similarity is selected at 410 as a first associated pair, two components of the selected pair (e.g., the caption / text summary and the image / content representation / data cluster) are removed from further consideration, and subprocessing 400 proceeds to 415.

[0061] At 415, a second or subsequent one of the text summary or caption is selected. The selection of the second or subsequent text summary or caption may be random, may be based on an order of user entry, may be selected in response to user input, or may be selected based on a previous user preference. Once the second or subsequent text summary or caption is selected, the pair containing the selected second or subsequent text summary and having the highest content similarity is selected at 420 as the second or subsequent associated pair, two components of the selected pair (e.g., the caption / text summary and the image / content representation / data cluster) are removed from further consideration, and subprocess 400 proceeds to 425.

[0062] At 425, it is determined whether the number of association pairs selected so far is less than a threshold. In some example embodiments, the threshold may represent a user setting of the number of captions / text summaries or images / content representations / data clusters to be generated in the generated media content book. In some example embodiments, the threshold may be set using a user interface (such as discussed in more detail below). Figure 6 and Figure 7 If the number of associated pairs selected so far is less than the threshold value (yes at 425), the sub-process 400 returns to 415 and continues from there. Conversely, if the number of associated pairs selected so far is not less than the threshold value (no at 425), the sub-process 400 ends and the main process for generating a content media book can continue.

[0063] Figure 5 A flowchart of a process 500 for generating a media content book according to another example embodiment of the present application is illustrated. Some aspects of the process 500 may be similar to those described above with reference to Figure 1 and Figure 2 The processes 100 and 200 discussed are similar, and therefore, redundant discussion may be omitted. Process 500 may be performed by a computing device in a computing environment, such as the one discussed below. Figure 8 An example computer device 805 of the illustrated example computing environment 800. Although the elements of the process 500 may be illustrated in a particular order, the example embodiments are not limited to the particular order illustrated. Without departing from the scope of the present application, the example embodiments may include actions that are sequenced into a different order as may be apparent to one of ordinary skill in the art or actions that may be performed in parallel or dynamically.

[0064] In process 500, multiple images or content representations may be clustered or divided into subsets of images or content representations based on content similarities between different images or content representations at 505. This may include extracting content features from the content representations or images to produce a feature embedding. Feature extraction may be performed using a neural network that has been trained using training data.

[0065] Again, in order to represent the content features of each content representation, subnetworks can be trained on different tasks to learn embeddings. For example, text embeddings can include using existing techniques (e.g., "word2vec") to learn vector encodings representing words that are extracted from the content representation as part of text extraction by extracting words from images using text recognition techniques (e.g., optical character recognition, etc.). The neural network can have a simple architecture that takes a large-sized single-hot representation of the vocabulary and then maps it with a very low-sized set of nodes to predict terms in the same single-hot representation in the final layer. This type of network can be trained by providing a large corpus of text and using a single-hot representation of the current term and training the network to predict subsequent terms in the text. The representation value can be used as a feature representation of the text term.

[0066] Further, multiple methods can be used to embed sentences or phrases. For example, neural methods that use the context of neighboring words or sentences can be effectively used by taking the word2vec representation discussed as input and embedding the caption into a low-dimensional embedding.

[0067] With respect to extracting content features that represent images, a deep convolutional neural network can be trained on a large corpus (e.g., "ImageNet"). In this network, images can be passed through a series of convolutional layers with max pooling and ReLU activation. After the convolutional layers, multiple flattened fully connected layers are applied for the final task of classifying the image, typically into one of the ImageNet classes. Once the network is trained on this data, the content representation can be passed through the network, and the penultimate layer output can be used as a feature representation, which is applied to clusters of the content representation to form data clusters of subsets of the content representation.

[0068] In some example embodiments, the image representation may be further enhanced with metadata reflecting additional information cues related to the capture of the image, such as timestamps, geographic coordinates, and camera settings, and the image representation may also use an automated system to enhance the metadata with information extracted from the image, such as the identity of people in the image or the projected aesthetic quality of the image.

[0069] Using the determined embeddings, content representations can be clustered before being assigned to a particular text summary caption. Each data cluster or subset of content representations can be associated with a caption based on the strength of aggregation of images associated with each group. In the case of capturing strong metadata cues such as temporal content representations, event-driven grouping that takes its position to ensure that images are not fragmented across multiple text summary-content representation sets can also be used.

[0070] After the data clusters are generated, user input is received by the system at 510. In process 500, the user input may include a user-provided text summary or caption that the user wants to exemplify. In some example embodiments, the text summary may be provided by extracting social media posts or microblog posts from a social media platform.

[0071] After receiving user input, the system can generate text summary-data cluster pairs at 515 by pairing each provided text summary with each possible data cluster or subset of the content representation.

[0072] At 520, a content similarity value is determined for each text summary-data cluster pair. Figure 1 As discussed in the example embodiments of , this may include extracting content features of the text summary and comparing with the aggregate content embeddings determined in 505 for each data cluster to produce a characterized embedding associated with both the caption and the data cluster. Feature extraction may be performed using a neural network that has been trained using training data. The training data may include a collection of images or photo books that have been manually generated by users of a captioned book editing or photo book generation platform that is accessible via a website, mobile application, or software installed on a local machine. For each text summary-data cluster pair, the extracted features (embeddings) may be concatenated together.

[0073] Again, in order to represent the content features of each data cluster and text summary, subnetworks can be trained on different tasks to learn embeddings. For example, text embeddings can include using existing techniques (e.g., "word2vec") to learn vector encodings representing words extracted from captions or text summaries. In some embodiments, as part of text extraction, text recognition techniques (e.g., optical character recognition, etc.) can also be used to extract words from images of each data cluster. The neural network can have a simple architecture that takes a large-size single-hotspot representation of the vocabulary and then maps it with the help of a very low-size set of nodes to predict the terms in the same single-hotspot representation in the final layer. This type of network can be trained by providing a large corpus of text and using a single-hotspot representation of the current term and training the network to predict subsequent terms in the text. For a user-provided text summary, each term can be passed through the network, and the representation of each term can be maintained in an intermediate low-size layer. The representation value can be used as a feature representation of the text term.

[0074] Further, multiple methods can be used to embed sentences or phrases. For example, neural methods that use the context of neighboring words or sentences can be effectively used by taking the word2vec representation discussed as input and embedding the caption into a low-dimensional embedding.

[0075] With respect to extracting content features representing images, a deep convolutional neural network can be trained on a large corpus (e.g., "ImageNet"). In this network, images can be passed through a series of convolutional layers with maximum pooling and ReLU activation. After the convolutional layers, multiple flattened fully connected layers are applied for the final task of classifying the image typically into one of the ImageNet classes. Once the network is trained on this data, user-provided images of various data clusters can be passed through the network, and the penultimate layer output can be used as a feature representation applied toward the selection task discussed below.

[0076] In some example embodiments, the image representation of each data cluster may be further enhanced with metadata reflecting additional information cues related to the capture of the image, such as timestamps, geographic coordinates, and camera settings, and the image representation may also use an automated system to enhance the metadata with information extracted from the image, such as the identity of the person in the image or the projected aesthetic quality of the image. Additionally, data indicating a particular user's past preferences or more past album or photo book generation selections may also be fed into the model to further improve feature extraction and image selection.

[0077] Once the content similarity is determined for each text summary-data cluster pair, the first text summary can be associated with the first data cluster based on the content similarity of the respective text summary-data cluster pairs at 525 to produce a first associated pair. Similarly, the second text summary can be associated with the second data cluster based on the content similarity of the respective text summary-data cluster pairs at 530 to produce a second associated pair. Example implementations of 525 and 530 can be implemented using the above respectively described in Figure 3 and Figure 4 525 and 530 may be repeated based on the content similarity determined for each text summary-data cluster pair until a desired number of associated pairs (e.g., three, four, five, etc. associated pairs of text summary-data clusters) are obtained.

[0078] Once the desired number of associated pairs is generated, an image album or photo book may be generated according to the user settings at 535, and the process may end. In some example embodiments, the user settings may be provided by a user interface (such as discussed in more detail below). Figure 6 and Figure 7 In some example embodiments, the user settings may be provided by settings associated with the image capture device and / or additional training data based on the user's previous albums or photo books.

[0079] Figure 6 and Figure 7 Illustrated are user interfaces (UIs) 600 and 700 that can be used to control a system according to an example embodiment of the present application. The UIs 600 and 700 can be displayed on a display device including, but not limited to, a computer monitor, a TV, a touch screen display of a mobile device, a laptop display screen, or any other display device that may be apparent to a person of ordinary skill in the art.

[0080] like Figure 6 As an example, UI 600 may include a display of content media representations or images 605a to 605c and text summaries 610a to 610c. UI 600 may also include: a book generation settings area 615, which may be used by a user to provide settings controls 640 to 650 for generating a book; and a text entry window 620, into which a user may type additional text summaries.

[0081] Within the book generation settings area 615, a settings control 640, illustrated as a text entry field with up and down clickable icons, can be used to select a minimum threshold number of representations to be used to generate the book. This threshold setting can be used in the above-mentioned processes (e.g., processes 100, 200, 500 and sub-processes 300, 400) to determine when a process or sub-process should be terminated. Further, a settings control 645, illustrated as a text entry field with up and down clickable icons, can be used to select a maximum threshold number of representations to be associated with any caption or text summary. This threshold setting can be used to define the size of a subset of content representations used to form a data cluster in the above-mentioned processes (e.g., processes 100, 200, 500 and sub-processes 300, 400) or limit their generation. In addition, a settings control 650, illustrated as a slider control, can be used to define the diversity of content media representations selected to be paired with different text summaries in the media book. For example, if the slider is used to define that a wide variety of content media representations should be used, the system can compare the similarities of content media representations that have been selected to be paired with different captions, and select alternative content media representations with greater differences to provide greater diversity in the generated album. Conversely, if the slider is used to define that a small variety of content media representations should be used, the system can compare the similarities of content media representations that have been selected to be paired with different captions, and select content media representations with greater similarities to reduce the diversity of the generated album.

[0082] The UI 600 may also include one or more buttons or icons 625 to 635, 655 that can be used by the user to control the book generation process. For example, button 625 can be used to reopen a previously entered text summary 610a to 610c via the text entry window 620 for editing. Conversely, button 630 can be used to submit the text entered into the text entry window 620 as the text summary 610a to 610c. Further, button 655 can be used to upload additional content media representations 605a to 605c for generating a media book. In addition, once the user uses the UI 600 to enter all desired content media representations 605a to 605c, text summaries 610a to 610c, and uses controls 640 to 650 to adjust the book generation settings, button 635 can be used to cause the system to generate a media book. Once the system generates the media book, the UI 600 can be transformed into Figure 7 Illustrated UI 700.

[0083] UI 700 includes some features similar to those illustrated in UI 600, and thus, similar reference numerals are used and redundant descriptions are omitted. As illustrated, UI 700 includes a book or photo book preview window 705 that displays text summary-content media representation pairs. For example, content media representation 605c is selected as content media representation 1 and is paired with text summary 610b selected as text summary 1. Similarly, content media representation 605a is selected as content media representation 2 and is paired with text summary 610c selected as text summary 2. Further, new text summaries 610d and 610e and new content media representations 605d and 605e are added in UI 700.

[0084] UI 700 may be used by a user to modify book generation settings using the book generation settings window 615 previously discussed above with respect to UI 600, and / or to modify the generated book by dragging different content media representations 605b, 605d, 605e or text summaries 610a, 610d, 610e into the book preview window 705. Once the user has modified the generated book using UI 700, button 735 may be used to regenerate the media book based on the changes specified by the user.

[0085] Sample computing environment

[0086] Figure 8 An example computing environment 800 is illustrated with an example computer device 805 suitable for use in some example embodiments. The computing device 805 in the computing environment 800 may include one or more processing units, cores, or processors 810, memory 815 (e.g., RAM, ROM, etc.), internal storage 820 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interfaces 825, any of which may be coupled on a communication mechanism or bus 830 for communicating information or embedded in the computing device 805.

[0087] The computing device 805 may be communicatively coupled to an input / interface 835 and an output device / interface 840. One or both of the input / interface 835 and the output device / interface 840 may be wired or wireless interfaces and may be detachable. The input / interface 835 may include any device, component, sensor, or interface (physical or virtual) that may be used to provide input (e.g., buttons, touch screen interfaces, keyboards, pointing / cursor controllers, microphones, cameras, braille, motion sensors, optical readers, etc.).

[0088] The output device / interface 840 may include a display, television, monitor, printer, speaker, Braille, etc. In some example embodiments, the input / interface 835 (e.g., user interface) and the output device / interface 840 may be embedded with or physically coupled to the computing device 805. In other example embodiments, other computing devices may function or provide functionality for the input / interface 835 and output device / interface 840 of the computing device 805. These elements may include, but are not limited to, well-known AR hardware inputs to allow a user to interact with an AR environment.

[0089] Examples of computing devices 805 may include, but are not limited to, highly mobile devices (e.g., smart phones, devices in vehicles and other machines, devices carried by people and animals, etc.), mobile devices (e.g., tablet computers, notebook computers, laptop computers, personal computers, portable televisions, radios, etc.), and devices that are not designed to be mobile (e.g., desktop computers, server devices, other computers, kiosks, televisions, radios, etc. having one or more processors embedded therein and / or coupled thereto).

[0090] The computing device 805 can be communicatively coupled to external storage 845 and network 850 (e.g., via I / O interface 825) to communicate with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. The computing device 805 or any connected computing device can be referred to as a server, client, thin server, general purpose machine, special purpose machine, or another label to perform its function or provide its service.

[0091] I / O interface 825 may include, but is not limited to, wired and / or wireless interfaces that use any communication or I / O protocol or standard (e.g., Ethernet, 802.11xs, universal system bus, WiMAX, modems, cellular network protocols, etc.) to communicate information to and / or from all connected components, devices, and networks in computing environment 800. Network 850 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0092] The computing device 805 may use and / or communicate using computer-usable or computer-readable media including transient media and non-volatile media. Transient media include transmission media (e.g., metal cables, optical fibers), signals, carrier waves, etc. Non-volatile media include magnetic media (e.g., optical disks and magnetic tapes), optical media (e.g., CD ROMs, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.

[0093] The computing device 805 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. Computer-executable instructions can be retrieved from transient media and stored on and retrieved from non-volatile media. Executable instructions can be derived from one or more of any programming language, scripting language, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others).

[0094] The processor 810 can be executed under any operating system (OS) (not shown) in a local or visual environment. One or more applications can be deployed, including a logic unit 855, an application programming interface (API) unit 860, an input unit 865, an output unit 870, a pairing unit 875, a feature extraction unit 880, a ranking unit 885, a pair association unit 890, and an inter-unit communication mechanism 895 for different units to communicate with each other, with the OS, and with other applications (not shown).

[0095] For example, the pairing unit 875, the feature extraction unit 880, the ranking unit 885, and the association unit 890 may implement Figures 1 to 5 One or more processes are shown. The units and elements described may vary in design, function, configuration or implementation and are not limited to the description provided.

[0096] In some example embodiments, when the API unit 860 receives information or executes instructions, it may communicate to one or more other units (e.g., logic unit 855, input unit 865, pairing unit 875, feature extraction unit 880, ranking unit 885, and pair association unit 890). For example, the pairing unit 875 may pair each caption or text summary with all images or content representations under consideration for an image album or photo book, and all pairs may be communicated to the feature extraction unit 880. Further, the feature extraction unit 880 may extract content features or embeddings from each text summary-content representation pair and provide the extracted content features or embeddings to the ranking unit 885. In addition, the ranking unit 885 may score the similarity of the text summary to the content representation in each pair and rank all pairs based on the score. The ranking of all pairs may be provided to the pair association unit 890, which selects or associates individual pairs for generating an image album or photo book, which may be output using the output unit 870.

[0097] In some cases, in some of the example embodiments described above, the logic unit 855 may be configured to control the flow of information between the units and direct the services provided by the API unit 860, the input unit 865, the pairing unit 875, the feature extraction unit 880, the ranking unit 885, and the association unit 890. For example, the flow of one or more processes or embodiments may be controlled by the logic unit 855 alone or in conjunction with the API unit 860.

[0098] Although some example embodiments have been shown and described, these example embodiments are provided to convey the subject matter described herein to people familiar with the art. It should be understood that the subject matter described herein can be implemented in various forms, not limited to the example embodiments. The subject matter described herein can be practiced without those specifically defined or described topics or without describing other or different elements or topics. Those familiar with the art will understand that changes can be made in these example embodiments without departing from the subject matter described herein as defined in the appended claims and their equivalents.

Claims

1. A method of associating a text summary with data representing media content, the method comprising: Generate a user interface with input areas; receiving, via the input area of ​​the user interface, a plurality of text summaries, each text summary representing an event; Pairing each received text summary with each of a plurality of data respectively representing media content through a neural network to generate a plurality of text-data pairs; associating a first text summary selected from the plurality of text summaries with a first data selected from the plurality of data based on similarities between content features extracted from the received respective text summaries and content features extracted from respective data in each of the plurality of text-data pairs; excluding the first data from the plurality of data, thereby creating a subset of data representing remaining media content; Based in part on the excluding, associating a second text summary selected from the plurality of text summaries with second data selected from the subset of data based on similarities between content features extracted from each received text summary and content features extracted from each data in each of the plurality of text-data pairs; as well as A media content book is generated based on the association between the first text summary and the first data and the association between the second text summary and the second data.

2. The method according to claim 1, wherein: The process of associating the first text summary with the first data is based on the fact that the similarity between the content feature extracted from the first data and the first text summary is the highest similarity of any text-data pair among the plurality of text-data pairs, and The process of associating the second text summary with the second data is based on the fact that the similarity between the content feature extracted from the second data and the second text summary is the second highest similarity of any text-data pair among the multiple text-data pairs.

3. The method according to claim 1, wherein: The process of associating the first text summary with the first data is based on the fact that the similarity of the content features extracted from the first data to the first text summary is the highest similarity of a first subset of text-data pairs selected from the plurality of text-data pairs, the first subset only including the following text-data pairs: in which the first text summary has been paired to each of the plurality of data, and Wherein, the process of associating the second text summary with the second data is based on the similarity between the content features extracted from the second data and the second text summary being the highest similarity of a second text-data pair subset selected from the multiple text-data pairs, the second subset only including the following text-data pairs: in this text-data pair, the second text summary has been paired to each data in the multiple data.

4. The method according to claim 1, wherein: The step of associating the first text summary with the first data comprises: clustering a first subset of the plurality of data based on content features extracted from respective data in the subset to form a first data cluster; Determining similarities between each data in the first data cluster and the first text summary; determining a first aggregate similarity score for the first data cluster based on the determined similarities between individual data in the first data cluster and the first text summary; and associating the first data cluster with the first text summary based on the first aggregate similarity score, and The step of associating the second text summary with the second data comprises: clustering a second subset of the plurality of data based on content features extracted from each data in the subset to form a second data cluster; determining similarities between each data in the second data cluster and the second text summary; determining a second aggregate similarity score for the second data cluster based on the determined similarities between individual data in the second data cluster and the second text summary; and The second data cluster is associated with the second text summary based on the second aggregate similarity score.

5. The method according to claim 1, wherein: Similarities between content features extracted from each received text summary and content features extracted from each data in each of the plurality of text-data pairs are calculated based on a user profile associated with a user requesting the text summary to be associated with data representing media content.

6. The method according to claim 5, wherein: Each text summary of the plurality of text summaries is generated based on a social media post associated with a user account associated with a user requesting that the text summary be associated with data representing media content.

7. The method according to claim 1, wherein: The similarity between the content features extracted from each received text summary and the content features extracted from each data in each of the multiple text-data pairs is calculated based on the following at least one capture setting, which is associated with a media capture device used to capture at least one data among the multiple data representing media content.

8. A non-volatile computer readable medium storing a program for causing a computer to execute a method of associating a text summary with data representing media content, the method comprising: Generate a user interface with input areas; receiving, via the input area of ​​the user interface, a plurality of text summaries, each text summary representing an event; Pairing each received text summary with each of a plurality of data respectively representing media content through a neural network to generate a plurality of text-data pairs; associating a first text summary selected from the plurality of text summaries with a first data selected from the plurality of data based on similarities between content features extracted from the received respective text summaries and content features extracted from respective data in each of the plurality of text-data pairs; excluding the first data from the plurality of data, thereby creating a subset of data representing remaining media content; Based in part on the excluding, associating a second text summary selected from the plurality of text summaries with second data selected from the subset of data based on similarities between content features extracted from each received text summary and content features extracted from each data in each of the plurality of text-data pairs; as well as A media content book is generated based on the association between the first text summary and the first data and the association between the second text summary and the second data.

9. The non-volatile computer readable medium of claim 8, wherein: The process of associating the first text summary with the first data is based on the fact that the similarity between the content feature extracted from the first data and the first text summary is the highest similarity of any text-data pair among the plurality of text-data pairs, and The process of associating the second text summary with the second data is based on the fact that the similarity between the content feature extracted from the second data and the second text summary is the second highest similarity of any text-data pair among the multiple text-data pairs.

10. The non-transitory computer readable medium of claim 8, wherein: The process of associating the first text summary with the first data is based on the fact that the similarity of the content features extracted from the first data to the first text summary is the highest similarity of a first subset of text-data pairs selected from the plurality of text-data pairs, the first subset only including the following text-data pairs: in which the first text summary has been paired to each of the plurality of data, and Wherein, the process of associating the second text summary with the second data is based on the fact that the similarity between the content features extracted from the second data and the second text summary is the highest similarity of a second text-data pair subset selected from the multiple text-data pairs, and the second subset only includes the following text-data pairs: in this text-data pair, the second text summary has been paired to each data in the multiple data.

11. The non-volatile computer readable medium of claim 8, wherein: The step of associating the first text summary with the first data comprises: clustering a first subset of the plurality of data based on content features extracted from respective data in the subset to form a first data cluster; Determining similarities between each data in the first data cluster and the first text summary; determining a first aggregate similarity score for the first data cluster based on the determined similarities between individual data in the first data cluster and the first text summary; and associating the first data cluster with the first text summary based on the first aggregate similarity score, and The step of associating the second text summary with the second data comprises: clustering a second subset of the plurality of data based on content features extracted from each data in the subset to form a second data cluster; determining similarities between each data in the second data cluster and the second text summary; determining a second aggregate similarity score for the second data cluster based on the determined similarities between individual data in the second data cluster and the second text summary; and The second data cluster is associated with the second text summary based on the second aggregate similarity score.

12. The non-transitory computer readable medium of claim 8, wherein: Similarities between content features extracted from each received text summary and content features extracted from each data in each of the plurality of text-data pairs are calculated based on a user profile associated with a user requesting the text summary to be associated with data representing media content.

13. The non-transitory computer readable medium of claim 12, wherein: Each text summary of the plurality of text summaries is generated based on a social media post associated with a user account associated with a user requesting that the text summary be associated with data representing media content.

14. The non-transitory computer readable medium of claim 8, wherein: The similarity between the content features extracted from each received text summary and the content features extracted from each data in each of the multiple text-data pairs is calculated based on the following at least one capture setting, which is associated with a media capture device used to capture at least one data among the multiple data representing media content.

15. A computer device configured to associate a text summary with data representing media content, the computer device comprising: A memory storing a plurality of data representing media content; as well as A processor that performs processing for visualizing recommended media content based on observed media content, the processing comprising: providing a user with an interface for submitting a plurality of text summaries, each text summary representing an event; Pairing each provided text summary with each of the plurality of data stored in the memory by a neural network to generate a plurality of text-data pairs; associating a first text summary selected from the plurality of text summaries with a first data selected from the plurality of data based on similarities between content features extracted from the received respective text summaries and content features extracted from respective data in each of the plurality of text-data pairs; excluding the first data from the plurality of data, thereby creating a subset of data representing remaining media content; Based in part on the excluding, associating a second text summary selected from the plurality of text summaries with second data selected from the subset of data based on similarities between content features extracted from the received respective text summaries and content features extracted from respective data in each of the plurality of text-data pairs; and Based on the association between the first text summary and the first data and the association between the second text summary and the second data, a media content book is provided.

16. The computer device of claim 15, wherein: The process of associating the first text summary with the first data is based on the fact that the similarity between the content feature extracted from the first data and the first text summary is the highest similarity of any text-data pair among the plurality of text-data pairs, and The process of associating the second text summary with the second data is based on the fact that the similarity between the content feature extracted from the second data and the second text summary is the second highest similarity of any text-data pair among the multiple text-data pairs.

17. The computer device of claim 15, wherein: The process of associating the first text summary with the first data includes: clustering a first subset of the plurality of data based on content features extracted from respective data in the subset to form a first data cluster; Determining similarities between each data in the first data cluster and the first text summary; determining a first aggregate similarity score for the first data cluster based on the determined similarities between individual data in the first data cluster and the first text summary; and associating the first data cluster with the first text summary based on the first aggregate similarity score, and The process of associating the second text summary with the second data includes: clustering a second subset of the plurality of data based on content features extracted from each data in the subset to form a second data cluster; determining similarities between each data in the second data cluster and the second text summary; determining a second aggregate similarity score for the second data cluster based on the determined similarities between individual data in the second data cluster and the second text summary; and The second data cluster is associated with the second text summary based on the second aggregate similarity score.

18. A computer program product, comprising computer instructions, which when executed by a processor perform the following operations: Generate a user interface with input areas; receiving, via the input area of ​​the user interface, a plurality of text summaries, each text summary representing an event; Pairing each received text summary with each of a plurality of data respectively representing media content through a neural network to generate a plurality of text-data pairs; associating a first text summary selected from the plurality of text summaries with a first data selected from the plurality of data based on similarities between content features extracted from the received respective text summaries and content features extracted from respective data in each of the plurality of text-data pairs; excluding the first data from the plurality of data, thereby creating a subset of data representing remaining media content; Based in part on the excluding, associating a second text summary selected from the plurality of text summaries with second data selected from the subset of data based on similarities between content features extracted from each received text summary and content features extracted from each data in each of the plurality of text-data pairs; as well as A media content book is generated based on the association between the first text summary and the first data and the association between the second text summary and the second data.

Images