Collage of extracted image segments

Abstract

Systems and methods are described for extracting image segments from images and including those extracted image segments in a collage. Origin information, such as source images from which extracted image segments are generated, source image locations, etc., are maintained as metadata such that interactions with the extracted image segments on the collage can be used to determine and / or return to the origin of the extracted image segments. The collage may be updated, shared, adjusted, etc., by the creator of the collage or other users.

Description

[Technical field]

[0001] (Priority Claim) This application claims priority to U.S. patent application Ser. No. 17 / 666,997, filed February 8, 2022, and entitled "Extracted Image Segments Collage," which in turn claims priority to U.S. Provisional Patent Application Ser. No. 63 / 291,276, filed December 17, 2021, and entitled "Extracted Image Segments Collage," the contents of each of which are incorporated by reference in their entirety herein. [Background technology]

[0002] As the amount of accessible digital content available to users and customers continues to grow, it becomes increasingly difficult to organize and maintain information about the digital content that is of interest to users and / or information discovered by users. For example, some systems allow users to maintain links or bookmarks to websites or specific web pages that the user has discovered. Other systems also allow users to save images of items discovered by the user. [Brief description of the drawings]

[0003] [Figure 1A] FIG. 1A illustrates a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1B] FIG. 1B illustrates a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1C] FIG. 1C illustrates a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1D] FIG. 1D illustrates a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1E] FIG. 1E depicts a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1F] FIG. 1F illustrates a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1G] FIG. 1G illustrates a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1H] FIG. 1H illustrates a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1I] FIG. 1I depicts a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1J] FIG. 1J illustrates a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1K] FIG. 1K illustrates a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1L] FIG. 1L illustrates a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1M]FIG. 1M illustrates a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1N] FIG. 1N illustrates a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Figure 1O] FIG. 1O depicts a graphical user interface, creating a collage of extracted image segments, and remixing the created collage according to the disclosed implementations. [Diagram 2] FIG. 2 is an exemplary collage process in accordance with the disclosed implementations. [Diagram 3] FIG. 3 is an exemplary image processing sub-process according to the disclosed implementation. [Figure 4] FIG. 4 is an exemplary image segment modification sub-process in accordance with the disclosed implementation. [Diagram 5] FIG. 5 is an exemplary collage conversion process according to the disclosed implementation. [Figure 6] FIG. 6 is an exemplary purchasable object collage process according to the disclosed implementations. [Figure 7A] FIG. 7A is a collage of exemplary purchasable objects in accordance with the disclosed implementations. [Figure 7B] FIG. 7B is an exemplary purchasable object detail page in accordance with the disclosed implementations. [Figure 8] FIG. 8 is an exemplary image segmentation deep neural network update process according to the disclosed implementations. [Figure 9] FIG. 9 illustrates an exemplary computing device in accordance with the disclosed implementations. [Figure 10] FIG. 10 illustrates an exemplary arrangement of components of a computing device such as that shown in FIG. [Figure 11] FIG. 11 is an illustration of an example implementation of a server system that may be used in various implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] Described herein are systems and methods for extracting image segments, referred to as extracted image segments, from images and including those image segments in a collage. Origin information, such as source image from which the extracted image segment was generated, source image location, etc., is maintained as metadata such that interactions with the extracted image segment on the collage can be used to determine and / or return to the origin of the extracted image segment. For example, if an extracted image segment on a collage originates from an e-commerce website, an address to the e-commerce website can be maintained in the metadata of the extracted image segment when it is generated and added to the collage.

[0005] The extracted image segment may be positioned anywhere on the collage presented on the user device. For example, the extracted image segment may be visually stacked relative to other extracted image segments of the collage, the extracted image segment may be rotated, the extracted image segment may be sized, etc. Similarly, in some implementations, the extracted image segment may be animated or otherwise distinguished as part of the collage when presented.

[0006] In some implementations, the objects represented in the extracted image segments may be available for purchase. For example, a seller of an item represented in an extracted image segment of a collage may be determined and associated with and / or identified in the metadata of the extracted image segment. Similarly, an indicator may be presented with the extracted image segment to indicate that the object represented in the extracted image segment may be purchased from a seller. A user, when viewing the collage, may interact with the extracted image segment and, for example, be redirected to an e-commerce website of the object seller to complete a purchase of the object. In other implementations, a user may interact with the extracted image segment and directly purchase the object represented in the extracted image segment.

[0007] As discussed further below, the image segment and / or extracted image segment may be any portion of an image and may correspond to an object represented in the image segment / extracted image segment. In some implementations, the image may be processed by a deep neural network ("DNN") that is trained to detect objects in the image and segment the image such that each object represented in the image corresponds to an image segment of the image. When viewing an image, the image segments determined for the image may be presented in a visually distinct manner from the image. A user may select an image segment, and pixels of the image corresponding to the selected image segment are extracted to generate an extracted image segment. Similarly, metadata may be included in the extracted image segment, such as a representation of the image, a location of the image, a link to a website from which the object represented by the extracted image segment can be purchased or acquired, additional information about the object, reviews of the object, a link to a second collage from which the image or extracted image segment was obtained, the popularity of the extracted image segment, an indication of the user who created the extracted image segment, etc.

[0008] 1A-1O depict a graphical user interface, creation of a collage of extracted image segments, and remixing of the created collage, according to the disclosed implementations. As shown, the disclosed implementations may be implemented in whole or in part on a user device 100, such as a mobile phone, smartphone, tablet, wearable, laptop, desktop, etc. In other implementations, portions of the disclosed implementations, such as image processing, image segmentation, and / or extraction of image segments, may be implemented on one or more remote computing resources and the images, image segments, extracted image segments, collage generation, etc., executed on the user device. As will be appreciated, any variations of the processing and / or other operations of the disclosed implementations may be implemented on one or more different devices. Similarly, the disclosed implementations may be provided, for example, as part of a social networking environment, an e-commerce environment, or any other form of interactive computing.

[0009] 1A, a user interface 111 is presented on a display of user device 100. In the illustrated embodiment, user interface 111 includes a number of images that can be viewed and optionally selected by a user through interaction with user device 100. In the example illustrated with respect to FIG. 1A, the user interface includes three columns of images. A user may view any number of images and select one or more images via user interface 111.

[0010] The images may be provided from a remote data store accessible to the user device 100, such as a social networking service, the Internet, or from memory on the user device, or may be generated from a camera or other imaging element of the user device. In general, images may be obtained from any source and utilized in the disclosed implementations.

[0011] In the illustrated example, a user selects image 112, for example, via physical interaction with a touch-based display of a user device. In response to the selection of image 112, and looking at FIG. 1B, an image segment of image 112 may be determined and presented with the image such that image segment 112-2 is distinct from the remainder of image 112-1. In the illustrated example, image segment 112-2 of image 112 (FIG. 1A) includes a wine bottle, and when presented by the user device, image segment 112-2 is presented such that it is visually distinct from the remainder of image 112-1. While the example discussed with respect to FIGS. 1A and 1B shows a wine bottle as image segment 112-2, in other implementations, other image segments, or all image segments, such as the image segment of wine glass 112-3 and the image segment of table 112-4, may be determined and visually presented such that the image segments are visually distinct.

[0012] In some implementations, additional images 124, image segments, and / or extracted image segments, such as images / extracted image segments visually similar to image segment 112-2, may also be presented on the user interface of device 100 in response to user selection of image 112. For example, in some implementations, the popularity or frequency of extracted image segments used in other collages by the same or other users may be monitored, and popular or trending extracted image segments may be presented to the user as additional images 124. Alternatively, or in addition, as another example, existing extracted images that are determined to be similar to other extracted images included in the user's collage and / or of potential interest to the user may be presented to the user as additional images 124. Other additional images 124 that may be presented include, but are not limited to, extracted image segments that allow for purchase of an object depicted in the extracted image segment, extracted image segments related to the extracted image segment of the collage and / or image segment, extracted image segments generated by the user who selected the image segment, etc.

[0013] In this example, a user interacting with the device selects image segment 112-2. Upon selecting image segment 112-2, pixels of image 112 that correspond to the selected image segment 112-2 are extracted from image 112 to generate an extracted image segment including the pixels. Additionally, as discussed further below, metadata may be included in the extracted image segment, including, but not limited to, an indication of the image, a location of the image, a link to a website from which the object represented by the extracted image segment can be purchased or obtained, additional information about the object, reviews of the object, a link to a second collage from which the image or extracted image segment was obtained, the popularity of the extracted image segment, an indication of the user who created the extracted image segment, and the like.

[0014] 1C, upon image segment selection and generation of extracted image segment, extracted image segment 132 is presented on the display of device 100 as part of collage 150. A user may interact with extracted image segment 132 included in collage 150. For example, a user may crop the extracted image segment, rotate the extracted image segment, and increase / decrease the size of the extracted image segment. In some implementations, an object represented in extracted image segment 132 may be further processed to determine a three-dimensional ("3D") mesh of the object so that a user can rotate the extracted image segment in three dimensions.

[0015] In addition to interacting with the extracted image segment 132, in some implementations the user may choose to lock the extracted image segment so that it cannot be further interacted with, transformed, its position / size cannot be changed, etc., through the selection of a lock control 133-1. Alternatively or additionally, the user may choose to generate a duplicate of the extracted image segment 132 through the selection of a duplicate control 133-2. Finally, if the user decides not to include the extracted image segment 132 in the collage, the user may remove or delete the extracted image segment by the selection of a delete control 133-3.

[0016] In the illustrated example, and referring to FIG. 1D , the user has adjusted the first extracted image segment 132 by reducing the size of the extracted image segment 132, rotating the extracted image segment 132, and moving the extracted image segment 132 to the left portion of the collage 150.

[0017] In addition to viewing the extracted image segments presented on the collage, additional information indicators 142, 144, 146, 148 may also be presented. Additional information indicator 142 may provide information indicating the number of extracted image segments included in the collage, which in this embodiment is one. Additional information indicator 144 may provide an opportunity for the user who created the collage to invite a second user to view the collage, for example, making the collage a collaborative collage (as discussed below) and chatting with the second user. Additional information indicator 146 may be a remix indicator that, when selected by the user or another user, remixes the presentation of the extracted image segments of the collage. Remixing may include adjusting the position, size, orientation, stack position, etc. of one or more extracted image segments of the collage. Additional information indicator 148 may be a duplicate indicator that, when selected by the user or another user, causes a duplicate (also called a child copy) of the collage to be generated. Similar to generating a duplicate collage in response to a conversion request by another user, as discussed below, a duplicate collage generated in response to selection of indicator 148 may appear visually the same, but the metadata of the collage and image segments may be updated to link back to or otherwise reference the collage from which it was generated.

[0018] Continuing with the current example, and now referring to FIG. 1E, after adding the first extracted image segment 132 to the collage, the user may again view any number of images on the user interface of device 100. As previously described, the user may scroll through any number and / or source of images and select an image, such as image 152. In this example, after selecting image 152, and referring again to FIG. 1F, image segment 152-2 corresponding to an object depicted in the image, in this example a floor lamp, may be determined and presented to the user via the display of device 100 such that image segment 152-2 is visually distinct from image 152-1. In the illustrated example, as part of the processing of the images to determine the object and generate the image segments, processing may also be performed to determine whether a portion of the object in the image segment is occluded by another object in image 152. In this example, it is determined that one of the legs 152-3 of the lamp depicted in image segment 152-2 is occluded. In such an embodiment, one or more image processing algorithms, such as an inpainting algorithm, may be used to determine the appearance, shape, and location of the occluded portions of the object and present the occluded portions of the object as part of image segment 152-2. Similarly, in some implementations, the user interface may also include modification controls 162 that may be selected by a user to modify the image segment by adjusting pixels of the image included in the image segment. Similarly, for occluded portions of the object that correspond to an image segment, modification controls 162 may be utilized by a user to inpaint or correct portions of the image segment that were determined through processing of the image segment.

[0019] 1G, upon selection of the image segment and generation of the extracted image segment, the extracted image segment 173 is presented on the display of the device 100 as part of the collage 150 along with the other extracted image segments of the collage, in this example, the extracted image segment 132. The user may interact with the extracted image segment included in the collage 150. For example, the user may crop the extracted image segment 173, rotate the extracted image segment, and increase / decrease the size of the extracted image segment. In some implementations, the object represented in the extracted image segment may be further processed to determine a 3D mesh of the object so that the user can rotate the extracted image segment in three dimensions. Similarly, if the collage 150 has more than one extracted image segment, a stack control 172 may be included that allows the user to move the extracted image segment up or down in a stack relative to the other image segments of the collage. For example, in the collage 150 presented in FIG. 1G, the second extracted image segment 173 is visually presented at the top of the stack such that it is presented on top of the first extracted image segment 132. A user may adjust the position of the extracted image segment by selecting the extracted image segment and interacting with the stack control 172, such as selecting stack control 172-2 to move up or stack control 172-1 to move down. For example, comparing FIG. 1G to FIG. 1H, in FIG. 1G, the second extracted image segment 173 is presented at the top of the stack and before the first extracted image segment 132. In response to a user selecting the second extracted image segment 173 and the move down control 172-1, now referring to FIG. 1H, the second extracted image segment is moved down in the stack and presented behind the first extracted image segment 132.

[0020] In the illustrated example, and referring to FIG. 1I , the user has adjusted the second extracted image segment 173 by reducing the size of the extracted image segment 173 and moving it to the bottom of the stack so that it is presented behind or below the first extracted image segment 132, and has moved it so that it is centered in the upper left corner of the collage 150, behind the first extracted image segment 132.

[0021] Continuing with the above example, and referring now to FIGURE 1J, a user has generated an image 182 using the imaging elements of device 100. In this example, image 182 includes a representation of a chair and other objects. Upon processing the image, as discussed below, an image segment 182-2 corresponding to the chair is determined and presented such that the image segment is visually distinct from the remainder of image 182-1, as illustrated in FIGURE 1K.

[0022] Similar to the above, the user interface may include modification controls 192 that may be selected by a user to change pixels of image 182 that are included or excluded from the image segment upon extraction. For example, in the example illustrated with respect to FIG. 1K, processing of image 182 to determine image segment 182-2 erroneously excluded pixels representing a rear leg 182-3 of a chair from image segment 182-2. Through interaction with modification controls 192, a user may select to add or remove pixels from the image segment. For example, with reference to FIG. 1L, a user may interact with modification controls 192 to either add pixels to the image segment by selecting add modification control 192-1 or remove pixels from the image segment by selecting remove modification control 192-2. In this example, a user selects add modification control 192-1 and, through interaction with the image, such as via a touch-based display of device 100, indicates which portion of the image, and therefore pixels, should be included or added to image segment 182-2. Specifically, in this example, the user selected to include pixels corresponding to the chair legs 182-3. Similarly, the user also selected to remove pixels corresponding to the space 182-4 between the chair seat and the back of the chair that were originally included in the image segment as a result of processing the image segment by selecting the remove modification control 192-2. Upon modifying the image segment to include or exclude pixels, in some implementations image processing of the image, e.g., portions of the image, such as the modified portions and / or portions corresponding to the object of interest, may be re-performed, as discussed further below.

[0023] In some implementations, rather than adjusting the image to include / exclude pixels of interest that are extracted as extracted image segments as discussed herein, the user may select to remove the object from all or a portion of the image. In such an embodiment, the depicted object may be removed from the image or a portion of the image, and an infill or inpainting process, as known in the art, may be utilized to assign pixel values ​​to pixels that previously represented the removed object. As a result, the image may be adjusted to appear as if the object was not included in the image. For example, as shown in FIG. 1K, if the user selects to remove a chair (object) from the image, an infill or inpainting process may be utilized to assign pixel values ​​such that the floor and walls of the room represented by the object are represented by the pixels that previously represented the chair.

[0024] After modifying the image segment, the user may select the image segment, and an extracted image segment 193 may be generated to include pixels of the image corresponding to the image segment 182-2 and metadata of the image segment. Similarly, the extracted image segment 193 is presented on the collage 150 with the other extracted image segments 132, 173, and the user may adjust the extracted image segment as discussed. Referring now to FIG. 1M, in this example, the user adjusts the third extracted image segment 193 to be positioned near the first extracted image segment 132 and the second extracted image segment 173, and in front of the second extracted image segment 173. Additionally, the image additional information 142 includes metadata for each of the three extracted image segments 132, 173, 193 that may be selected and viewed by the user or other viewers of the collage. Similarly, the image additional information 142 indicates that there are three extracted images included in the collage 150.

[0025] The user may go through a process of extracting image segments and including the extracted image segments on the collage 150 for any number of extracted image segments, each of which may be placed anywhere on the collage. Similarly, in some implementations, the user may draw or write on the collage and / or may choose to animate one or more of the extracted image segments. Referring to FIG. 1N, a collage is illustrated that includes six extracted image segments 132, 173, 193, 195, 196, 197 and a typed note 198 ("MY FIRST COLLAGE") added by the user. Similarly, the user has selected animation 170-2 for the star extracted image segment 196 such that the star extracted image segment has a blinking animation effect. Additionally, the user has added animation to the cat extracted image segment 197 including both movement 170-3 and sound (e.g., "meow") animation 170-1 of the cat extracted image segment 197. Any of a variety of forms of animation may be added to the extracted image segments included in the collage. For example, animation may include, but is not limited to, color changes, movement, blinking, sound, haptics, and the like.

[0026] FIG. 1O is an example diagram of the collage 150 shown in FIG. 1N after selecting a remix control 146 in accordance with disclosed implementations.

[0027] As shown, in response to selection of the remix control 146, the positions of the extracted image segments 132, 173, 193, 195, 196, 197 on the collage and relative to one another are rearranged or remixed. In some implementations, the rearrangement or remix of the extracted image segments may be random. In other implementations, the rearrangement may be based on, for example, the popularity of the extracted image segments, user preferences, cross pattern configurations, layouts, etc.

[0028] FIG. 2 is an example collage process 200 in accordance with the disclosed implementations.

[0029] The exemplary process 200 begins upon receiving an image, as at 202. As mentioned above, the image may be from any source, such as a camera or other imaging element, from a website, from photos stored in memory of the user device or in memory accessible by the user device (local or remote), from a video frame from a video, etc. Similarly, in some implementations, the image received by the exemplary process may be an extracted image segment. For example, in some implementations, the popularity or frequency of extracted image segments used in other collages by the same or other users is monitored, and popular or trending extracted image segments are presented to the user for selection and inclusion in the collage. Alternatively, or in addition, as another example, existing extracted image segments that are determined to be similar to other extracted image segments included in a collage by the user and / or of potential interest to the user may be presented and / or selected as images by the user.

[0030] It may then be determined, as at 204, whether an area of ​​interest is indicated by a user. For example, in addition to receiving an image, a user may indicate an area or portion of the image that is of interest to the user, for example, through interaction with the image. If it is determined that an area of ​​interest is indicated, a portion of the image that is included in the indicated area of ​​interest is provided as an image, as at 206. If it is determined that an area of ​​interest is not provided, or after providing a portion of the image that is included in the indicated area of ​​interest as an image, an exemplary image processing sub-process 300 is performed on the image, as at 300. The exemplary image processing sub-process 300 is discussed in more detail below with respect to FIG. 3. As discussed below, the image processing sub-process 300 generates and returns one or more image segments of the image, each image segment corresponding to an object represented in the image. In some implementations, the image processing sub-process 300 may be performed independently of the exemplary process 200, such that image segments are already determined for and associated with the image. In such an embodiment, when an image is received, the image segments already determined for the image may be utilized without re-processing the image in the image processing sub-process.

[0031] One or more of the image segments returned by the image processing sub-process may then be presented to the user such that the image segments are distinct from the remainder of the image, as at 208. Examples of presenting the image segments such that the image segments are distinct from the rest of the image are shown in FIGS. 1B, 1F, 1K, and 1L and discussed above. In some implementations, all image segments determined for an image may be presented to the user as distinct from the rest of the image. In other examples, only a subset (e.g., 3-5) of the image segments determined for an image may be presented to the user as distinct from the rest of the image. In still other examples, only a single image segment, such as a primary or central image segment of an image, may be presented to the user as distinct from the rest of the image.

[0032] After presenting the image segment, a determination is made as to whether modifications to the presented image segment have been received, as at 210. As described above with respect to Figures 1F, 1K, and 1L, a user may interact with the image segment to modify the image segment to include / exclude pixels from the image segment. For example, if initial processing of the image incorrectly included / excluded a portion of an object depicted in the image segment, a user may interact with the image segment to adjust the included / excluded pixels in the image segment such that the image segment corresponds to the object of the image segment.

[0033] If it is determined that modifications to the image segment are received, then an exemplary image segment modification sub-process may be performed, such as 400 (FIG. 4). The exemplary image segment modification sub-process 400 is discussed further below with respect to FIG. 4. Once the image segment modification sub-process 400 is completed, the exemplary process 200 returns to block 208 and continues.

[0034] If it is determined that no modifications to the image segment have been received, then a determination is made as to whether a selection of an image segment for the image has been received, as at 212. If a selection of an image segment has not been received, the exemplary process 200 returns to block 210 and continues. If a selection of an image segment is received, then pixel data and corresponding metadata for the selected image segment are extracted and used to create an extracted image segment for the selected image segment, as at 216. As further discussed above, the metadata may include, but is not limited to, an indication of the image from which the image segment was extracted, the location of the image from which the image segment was extracted, a link to a website where the object represented by the extracted image segment can be purchased or acquired, additional information about the object, reviews of the object, a link to a second collage from which the image or extracted image segment was obtained, the popularity of the extracted image segment, an indication of the user who created the extracted image segment, and the like. The metadata included in the extracted image segment may be used for attribution information for the extracted image segment, enabling purchase of the object represented in the extracted image segment.

[0035] The extracted image segment may also be presented on a collage, as at 218. If this is the first extracted image segment of the collage, the extracted image segment may be presented on a blank collage. If other extracted image segments are already included on the collage, the extracted image segment may be presented initially in the center of the collage to allow the user to adjust the size, orientation, position, etc. of the image within the collage.

[0036] After presenting the extracted image segments on the collage, a determination is made as to whether any adjustments to the extracted image segments have been received, as at 220. The adjustments may include, for example, adjusting the size, position, orientation, and / or rotation of the extracted image segments, and / or animation of the extracted image segments.

[0037] If it is determined that an adjustment to the extracted image segment has been received, the extracted image segment is adjusted according to the received adjustment, as at 222. After adjusting the extracted image segment, the exemplary process 200 returns to decision block 220 and continues. If it is determined that an adjustment to the extracted image segment has not been received, a collage of the extracted image segments is presented, as at 224, and it is determined whether another extracted image segment is to be added to the collage, as at 226. As discussed, any number of extracted image segments may be added to the collage. If it is determined that another extracted image segment is to be added to the collage, the exemplary process 200 returns to block 202 and continues receiving another image. If it is determined that another extracted image segment is not to be added to the collage, as at 228, the exemplary process 200 is complete.

[0038] FIG. 3 is an exemplary image processing sub-process 300 according to the disclosed implementation.

[0039] The example sub-process 300 begins at 302 by segmenting the image. Any of a variety of segmentation techniques may be used, such as circle filling algorithms, superpixels, etc. The segments may then be processed at 304 to remove background portions of the image from consideration. The determination of background segments may be done, for example, using a combination of attentive constraints (e.g., salient objects are more likely to be in the center of the image) and uniqueness constraints (e.g., salient objects are more likely to be different from the background). In one implementation, each segment (S i ), a uniqueness constraint may be computed using a combination of color, texture, shape, and / or other feature detection. The pairwise Euclidean distance of all segment pairs: L2(S i , S j )too,

number

number

number

[0040] Next, one or more of the segments S', which is a subset of S, are U(s)-A( s )>t, where t is a threshold that is set manually or learned from data. The threshold t may be any defined number or amount that is utilized to distinguish a segment as background information or a potential object. Alternatively, where

number

number

number

[0041] Returning to FIG. 3, once the background segments have been removed, at 306, the objects remaining in the image are determined. The objects remaining in the image may be determined by computing a score for each possible hypothesis for the location of the object, for example, using a sliding window approach. Each segment may be processed to determine potentially matching objects using approaches such as enhanced selection with Haar-like wavelets or multiple part-based models. For example, an image vector may be determined for the segment and compared to stored information about the object. Based on the image vector and the stored information, a determination may be made as to how similar the image vector is to the stored information of a particular object.

[0042] The DNN may implement the sliding window approach N times, each time using a different trained object classifier (e.g., person, bag, shoe, face, arm, etc.). After determining the hypotheses for each object classifier, the output is the set of best hypotheses for each object class. Because objects generally do not appear randomly in an image (e.g., eyes and noses generally appear together), position-sensitive constraints may also be considered. For example, the position of the root object (e.g., person) may be defined as W(root), and each geometric constraint for each object k, relative to each other, is a 6-element vector

number

number

number

number

number

[0043] To optimize this function, the location of an object in the image may be determined at 308. For example, the center of a root object (e.g., a person) in the image may be marked as (0,0), and the locations of other objects in the processed image are shifted relative to the root object. Then, Θ i Apply a linear support vector machine (SVM) with parameters. The input to the SVM is D train (Θ i ). Other optimization approaches such as linear programming, dynamic programming, convex optimization, etc. may also be used, either alone or in combination with the optimization discussed herein. The training data D train (Θ k ) may be collected by having a user place bounding boxes over both the entire object and landmarks. Alternatively, semi-automated approaches such as face detection algorithms, edge detection algorithms, etc. may be utilized to identify objects. In some implementations, other shapes such as ellipses, ovals, and / or irregular shapes may be used to represent objects.

[0044] Finally, as at 310, an image segment for each detected object is maintained. As will be appreciated, the exemplary sub-process 300 of processing an image may be performed by a trained DNN that processes the image to generate image segments corresponding to objects represented in the image. For example, a DNN, such as a convolutional neural network, may be trained, for example, using labeled and / or unlabeled data to process an input image and output one or more image segments of the image that correspond to objects detected in the image. Similarly, as the image segments are refined by a user, as discussed further below, those refined image segments and corresponding images may be utilized as additional labeled training data to continue training the DNN, thereby further improving the accuracy of the DNN based on user-provided input.

[0045] FIG. 4 is an example image segment modification sub-process 400 according to the disclosed implementation.

[0046] The exemplary process 400 begins, as at 402, by adjusting an image segment based on user input, such as via a touch-based display, to include and / or exclude pixels from the image, thereby generating an adjusted image segment. For example, as described above with respect to FIGS. 1K and 1L, a user may interact with the disclosed implementations to add pixels to the image segment, remove pixels from the image segment originally determined and presented to the user, and / or remove objects from the image by replacing pixel values ​​using an infill or inpainting process. For example, as illustrated in FIGS. 1K and 1L, the image segment 182-2 originally determined for a chair excluded pixels of the image representing the rear leg 183-3 of the chair. Similarly, the original image segment included pixels of the image corresponding to the space 182-4 between the seat of the chair and the back of the chair. The user may interact with the presentation of the image to include pixels corresponding to the rear leg 182-3 of the chair in the image segment 182-2 and exclude pixels corresponding to the space 182-4 between the back of the chair and the seat of the chair.

[0047] The adjusted image segment may then be processed again to determine whether to identify an object contained in the adjusted image segment, as at 404. If it is determined that the adjusted image segment is processed to determine an object contained in the image segment, the example image processing sub-process 300 described above with respect to FIG.

[0048] After processing the adjusted image segment, or if it is determined that the adjusted image segment will not be processed again, the metadata of the adjusted image segment is updated to include / exclude the display of the pixels in / from the metadata, as at 406. Similarly, if the image is to be processed again, the information obtained from the example process 300 may be updated in the image segment's metadata. Finally, the adjusted image segment, or data corresponding to the adjusted image segment, is returned, as at 408.

[0049] FIG. 5 is an example collage conversion process 500 in accordance with the disclosed implementations.

[0050] The exemplary collage conversion process 500 begins by presenting a collage including one or more extracted image segments, as at 502. For example, a collage such as the collage 150 shown and described above with respect to FIG. 1N, including six image segments and a text input, may be presented to a user. In some implementations, the collage may be designed as private, such that only the user who created the collage can view and / or navigate the collage. In other implementations, the user may designate the collage as collaborative and invite other users to view and / or collaborate on the collage. Collaboration between the user and the collage may be real-time or near real-time, such that each of the users collaborating on the collage can see the changes to the collage and / or chat about the collage / changes to the collage. In other examples, collaboration may be incremental, such that a second user transforms the collage and the first user can later view the collage and see the changes made by the second user.

[0051] In yet other embodiments, a user may make a collage public so that any user can view the collage. A collaborative collage is one in which the collage can be modified by invited users, or any other user other than the creator of the collage if allowed by the collage creator.

[0052] After presenting the collage, a transformation request may be received to transform one or more aspects of the collage, as at 504. The transformation request may be any input for transforming one or more aspects of the collage, such as an extracted image segment of the collage. For example, the transformation request may include, but is not limited to, a request to remix the visual arrangement and presentation of the extracted image segments of the collage, a request to add an extracted image segment to the collage, a request to remove an extracted image segment from the collage, a request to adjust the size, shape, and / or position of the extracted image segments of the collage, a request to add, remove, or change animations of the extracted image segments of the collage, etc.

[0053] In response to receiving the conversion request, a determination is made as to whether the conversion request is from the creator of the collage (first user), as at 506. For example, a user identifier, or a user identifier associated with an application running on a user device used to create the collage, may be indicated as the creator of the collage. If the user utilizes another user device or accesses a user account associated with the same user device, user or user account, the conversion request may be determined to have been from the creator of the collage.

[0054] If it is determined that the request is from the creator of the collage, the collage is transformed according to the transformation request, as at 508. If it is determined that the transformation request is not from the creator of the collage, it is determined whether the collage is a collaborative collage, as at 509. As described above, the creator of the collage may indicate that the collage is collaborative so that other users may transform the collage. In such an embodiment, the collage may be transformed by the user and / or other users, and those transformations to the collage may be presented to the user and / or other users. If it is determined that the collage is from the creator of the collaborative collage, the collage is transformed according to the transformation request, as at 508. In some implementations, the user may specify which other users may transform the collage, such that the collage is considered a collaborative collage only for those particular users. For any other users who send transformation requests to a collage, a duplicate collage may be generated, as discussed below, to which the transformation requests may be applied such that the transformations do not affect the collage generated by the user.

[0055] If the collage is determined to be not a collaborative collage or not a collaborative collage for the user submitting the conversion request, a duplicate collage is generated for another user, referred to herein as the second user, as at 510. The duplication process may include the same extracted image segments in the same positions, orientations, sizes, etc. as the collage, such that the user converting the collage cannot determine differences between the duplicate collage and the collage. However, the metadata of the collage and each extracted image segment may be updated to indicate that the collage is a duplicate collage and includes information, links, and / or other references to the collage from which the duplicate was generated, as at 512. Similarly, the metadata of each extracted image segment may be updated to indicate the original collage as the source of the extracted image segment. Such information may be in addition to any source information already included in the metadata of the original collage and / or the extracted image segments.

[0056] Finally, the duplicate collage may be transformed in accordance with the received transformation request, as at 514. The duplicate collage becomes another collage maintained by the system, the second user is identified as the creator of the duplicate collage, and there are links or other references maintained between the duplicate collage, the original collage, and any other source information of the collage and / or extracted image segments included in the duplicate collage. Similarly, the second user may transform the duplicate collage without transforming the original collage. Additionally, the second user may also designate the duplicate collage as a private collage, a duplicate collage, etc., as if the second user were the original creator of the duplicate collage.

[0057] FIG. 6 is an exemplary purchasable object collage process 600 in accordance with the disclosed implementations.

[0058] The exemplary process 600 begins, as at 602, by determining objects represented by extracted image segments included in the collage. For example, any of a number of image processing algorithms or DNNs may be utilized to process the images and detect objects or object types represented in the images. Alternatively, or in addition, metadata about the extracted image segments may be utilized to determine the objects represented in the extracted image segments. For example, if the extracted image segments are originally obtained from a website, the metadata for the extracted image segments may include an indication of the objects represented in the extracted image segments.

[0059] In addition to determining the objects represented in the extracted image segments, one or more sellers of the objects may be determined, as at 604. For example, if the extracted image segments were originally obtained from a website, such as an e-commerce website, metadata of the extracted image segments may indicate a seller of the objects. In other embodiments, the seller of the objects may provide information, such as a catalog, that indicates the objects offered for sale by that seller. In yet other embodiments, the websites of the sellers may be processed to determine the objects offered for sale by those sellers, and information that is used to determine one or more sellers of the objects represented in the extracted image segments. In yet another embodiment, a seller or other user may provide an indication of a seller of the objects represented in the image segments.

[0060] Each determined merchant may then be associated with the extracted image segment, as at 606. For example, if the merchant corresponds to an e-commerce website, the object detail page may be associated with the extracted image segment, thereby indicating the merchant of the object.

[0061] In response to determining one or more vendors of the objects depicted in the image segments, purchasable indications may be presented with the extracted image segments as part of the collage, as at 608. For example, FIG. 7A is an illustration of a collage 740 presented on a user device 700 including purchasable indicators, in accordance with disclosed implementations. In this example, the collage 740 includes five extracted image segments 743-1, 743-2, 743-4, 743-5, 743-6, and a typed text input 743-3 for "My Christmas List." The extracted image segments of the collage 740 may be processed by the exemplary process 600, and a determination may be made that the extracted image segments 743-1 (a bicycle), 743-2 (a cowboy hat), and 743-5 (a book) correspond to purchasable objects. Thus, purchasable indications 745-1, 745-2, and 745-3 are presented next to their respective extracted image segments. In this example, the sweater object represented by extracted image segment 743-4 may have previously been indicated as available for purchase and may be indicated as purchased through the presentation of purchased indicator 747. For example, if the user who created the collage purchases the object represented by an extracted image segment, the available for purchase indicator for that extracted image segment may be replaced with purchased indicator 747, indicating that the item has been purchased.

[0062] In other examples, a collage may be created by a first user and shared with other users in accordance with disclosed implementations to indicate items that the first user would like to receive, e.g., Christmas gifts, birthday gifts, wedding gifts, etc. In such examples, the collage may be shared with one or more other users. One or more other users may interact with the collage 740 and, optionally, purchase items that correspond to extracted image segments included in the collage. In such examples, once the item is purchased or otherwise obtained, the available for purchase indicator may change to a purchased indicator, thereby indicating to other users that the item has already been purchased for the first user.

[0063] Returning to Figure 6, after presenting the collage with a purchase availability indicator with a corresponding extracted image segment, a determination may be made as to whether another extracted image segment of the collage remains to be processed by the exemplary process 600, as at 610. If it is determined that additional extracted image segments of the collage remain, then the next extracted image segment is selected, as at 612, and the exemplary process 600 returns to block 602 and continues. If it is determined that additional extracted image segments of the collage do not remain, then the exemplary process 600 is complete, as at 614.

[0064] In some implementations, when a user selects one of the extracted image segments that is indicated as available for purchase, such as extracted image segment 743-1, a purchasable object details page corresponding to the object represented by the extracted image segment may be presented.

[0065] 7B is an exemplary purchasable object details page in accordance with disclosed implementations. In particular, in response to a user selecting extracted image segment 743-1 of a bicycle, a purchasable object details page 755 is presented that includes additional information about the object represented by the selected extracted image segment, in this example, extracted image segment 743-1. For example, purchasable object details page 755 may include an indication of the price 755-1 of the object, a delivery time frame 755-2 for the object upon purchase, a seller 755-3 of the object, etc. Additionally, purchasable object details page 755 may include a purchasing control, such as a "Buy" button 757, that when selected enables the purchase of the object represented by extracted image segment 743-1 from seller 755-3, in this example, Company A.

[0066] FIG. 8 is an example image segmentation deep neural network update process 800 according to disclosed implementations. As mentioned above, in some implementations, the DNN used to determine image segments of an image may be continuously or periodically updated as image segments are adjusted by a user to include / exclude pixels in those image segments. In some implementations, when image segments are adjusted on a user device, the adjusted image segments may be sent to a remote computing resource and compiled with other user feedback (other image segment adjustments) and the adjusted image segments used for ongoing training to update the DNN. Once the DNN is updated, the updated DNN may be sent to the user devices for operation on those user devices. Alternatively, the updated DNN may reside on one or more remote computing resources and operate on those remote computing resources.

[0067] The exemplary process 800 begins by initially training a DNN to generate one or more image segments for an input image, as at 802. In some implementations, the DNN may be trained to perform the image processing subprocess 300 described above with respect to Figure 3. In other implementations, other training techniques or processes may be used to train the DNN to receive an input image and determine one or more image segments that correspond to objects depicted in the input image.

[0068] At some point after the DNN is initially trained, one or more adjusted image segments may be obtained based on user input that caused adjustments to the image segments originally determined by the DNN, as at 804. With a significantly larger set of users, a large set of adjusted image segments may be received as different users interact with the images and image segments determined and presented according to the disclosed implementations.

[0069] The trained image segments and corresponding images may be utilized as labeled training data for the DNN, and thus the trained image segments may be used to update the DNN, as at 806.

[0070] FIG. 9 illustrates an exemplary user device 900 that may be used in accordance with various implementations described herein. In this example, the user device 900 includes a display 902 and, optionally, at least one input component 904, such as a camera, on the same side and / or opposite side of the device as the display 902. The user device 900 may also include an audio transducer, such as a speaker 906, and optionally a microphone 908. In general, the user device 900 may have any form of input / output component that allows a user to interact with the user device 900. For example, various input components for enabling user interaction with the device may include a touch-based display 902 (e.g., resistive, capacitive, Interpolated Force Sensing Resistor (IFSR)), a camera (such as gesture tracking), a microphone, a global positioning system (GPS), a compass, or any combination thereof. One or more of these input components may be included on the user device or may otherwise communicate with the user device. Various other input components and combinations of input components can be used within a variety of implementations that should be apparent in light of the teachings and suggestions contained herein.

[0071] To provide various functions described herein, FIG. 10 illustrates an exemplary set of basic components 1000 of a user device, such as the user device 900 described with respect to FIG. 9 and discussed herein. In this example, the device includes one or more processors 1002 for executing instructions that may be stored in at least one memory device or element 1004. As would be apparent to one skilled in the art, the device may include many types of memory, data storage, or computer-readable storage media, such as a first data storage for program instructions for execution by the one or more processors 1002. Removable storage memory may be available to share information with other devices, etc. The device typically includes some type of display 1006, such as a touch-based display, electronic ink (e-ink), organic light-emitting diode (OLED), liquid crystal display (LCD), etc.

[0072] As discussed, in many implementations, the device includes at least one image capture element 1008, such as one or more cameras capable of capturing images of objects in the vicinity of the device. The image capture element may include or be based at least in part on any suitable technology, such as a CCD or CMOS image capture element having a determined resolution, focal range, visibility, and capture rate. The device may include at least one application component 1010 for performing implementations discussed herein, such as generating a collage. The user device may be in constant or intermittent communication with one or more remote computing resources, and may exchange information with the remote computing systems, such as collages, extracted image segments, transformed image segments, metadata, updated DNNs, etc., as part of the disclosed implementations.

[0073] The device may also include at least one location component, such as GPS, NFC location tracking, Wi-Fi location monitoring, etc. Location information obtained by the location component may be used in conjunction with various implementations discussed herein, for example, as a factor in determining a seller of an object represented in an extracted image segment. For example, if a user is located at store A and generates an extracted image segment from an image generated by the image capture element 1008 of the user device while located at store A, the location information may be used as a factor in determining a seller of an object represented in the extracted image segment.

[0074] The user device may also include a DNN 1012 operable to receive an image as an input and determine one or more image segments that correspond to objects depicted in the input image, as discussed herein. Similarly, the user device may also include a collage management component 1014 that, for example, maintains collages created and / or viewed by a user of the user device, extracted image segments, etc., and / or performs some or all of the implementations discussed herein.

[0075] An exemplary user device may also include at least one additional input device capable of receiving conventional input from a user. This conventional input may include, for example, push buttons, a touch pad, a touch-based display, a wheel, a joystick, a keyboard, a mouse, a trackball, a keypad, or any other such device or element by which a user may send input to the device. These I / O devices may also be connected by wireless, infrared, Bluetooth, or other links in some implementations. However, in some implementations, such devices may not include any buttons and may be controlled solely by touch input (e.g., a touch-based display), voice input (e.g., speech), or a combination thereof.

[0076] 11 is a diagram of an example implementation of a server system 1100, such as a remote computing resource, that may be used with one or more of the implementations described herein. The server system 1100 may include one or more processors 1101, such as one or more redundant processors, a video display adapter 1102, a disk drive 1104, an input / output interface 1106, a network interface 1108, and a memory 1112. The processors 1101, the video display adapter 1102, the disk drive 1104, the input / output interface 1106, the network interface 1108, and the memory 1112 may be communicatively coupled to each other by a communication bus 1110.

[0077] The video display adapter 1102 provides display signals to a local display and allows an operator of the server system 1100 to monitor and configure the operation of the server system 1100. The input / output interface 1106 similarly communicates with external input / output devices not shown in FIG. 11, such as a mouse, keyboard, scanner, or other input and output devices that may be operated by an operator of the server system 1100. The network interface 1108 includes hardware, software, or any combination thereof for communicating with other computing devices. For example, the network interface 1108 may be configured to provide communication between the server system 1100 and other computing devices, such as the user equipment 900.

[0078] The memory 1112 generally includes random access memory (RAM), read only memory (ROM), flash memory, and / or other volatile or fixed memory. The memory 1112 is shown storing an operating system 1114 for controlling the operation of the server system 1100. The server system 1100 may also include a trained DNN 1116 as discussed herein. In some implementations, the DNN may determine object segments on the server. In other implementations, the DNN 1012 (FIG. 10) may determine image segments on the user device. In yet other examples, the DNN may reside on both the server and each user device.

[0079] The memory 1112 further stores program code and data for providing network services that enable user devices 900 and external sources to exchange information and data files with the server system 1100. The memory 1112 may also include a collage management application 1118 that maintains collages and / or collage information for different users utilizing the disclosed implementations. The collage management application 1118 may communicate with a data store manager application 1120 to facilitate data exchange and mapping between the data store 1103, user devices such as the user device 900, external sources, and the like.

[0080] As used herein, the term "data store" refers to any device or combination of devices capable of storing, accessing, and retrieving data, and may include any combination and number of data servers, databases, data storage devices, and data storage media in any standard, distributed, or clustered environment. The server system 1100 may include any suitable hardware and software for integrating with the data store 1103 as necessary to execute aspects of one or more applications for user devices 900, external sources, and the like.

[0081] The data store 1103 may include a number of separate data tables, databases, or other data storage mechanisms and media for storing data related to a particular aspect. For example, the data store 1103 may include digital items (e.g., images) and corresponding metadata about those items (e.g., image segments, popularity, source). Collage data and / or user information and / or other information may be stored in the data store as well.

[0082] Of course, there may be many other aspects that may be stored in the data store 1103, which may be stored in any of the mechanisms listed above, or in any additional mechanisms in the data store, where appropriate. The data store 1103, through logic associated therewith, may be operable to receive instructions from the server system 1100 and, in response thereto, retrieve, update, or otherwise process data.

[0083] The server system 1100, in one implementation, is a distributed environment, using one or more computer networks or direct connections, utilizing several computer systems and components interconnected via communication links. However, it will be understood by those skilled in the art that such a system may operate equally well in a system having fewer or more components than those illustrated in Figure 11. Thus, the depiction of Figure 11 should be taken as being exemplary in nature and not limiting on the scope of the present disclosure.

[0084] Implementations disclosed herein may include a computer-implemented method including one or more of: processing a first image to determine a first image segment that corresponds to less than all of the first image and corresponds to an object represented in the first image; presenting the first image and the first image segment on a user device such that the first image segment is visually distinct from the first image; and, in response to a selection of the first image segment, forming a first extracted image segment including at least pixel data corresponding to pixels of the first image segment and metadata indicative of at least one of the first image or a source location of the first image. The computer implementation may also include one or more of presenting the first extracted image segment as part of a collage including at least a second extracted image segment representing a second object extracted from the second image, the first extracted image segment may be transformed relative to the second extracted image segment, the first extracted image segment may be transformed relative to a size of the first extracted image segment when presented on the collage, and the first extracted image segment may be transformed relative to a location of the first extracted image segment when presented on the collage.

[0085] Optionally, the computer-implemented method further includes receiving a modification of the first image segment to modify which pixels of the first image are included in the first image segment. Optionally, the metadata further includes one or more of a link to a website from which the object can be purchased, additional information about the object, reviews of the object, a link to a second collage including the first extracted image segment, a popularity of the first extracted image segment, a comment corresponding to the first extracted image segment, a rating of the first extracted image segment, an indication of other collages including the first extracted image segment, or an indication of a user who created the first extracted image segment. Optionally, processing the first image may include processing the first image to determine a plurality of image segments, the first image segment being included within the plurality of image segments, and / or presenting includes presenting each of the plurality of image segments on the user device such that each of the plurality of image segments is visually distinct from the first image. Optionally, the computer-implemented method further includes receiving an indication of an area of ​​interest in the first image, and processing includes processing a portion of the first image within the area of ​​interest to determine the first image segment. Optionally, the processing of the first image may take place on the user device.

[0086] Implementations disclosed herein may include a computing system including one or more processors and a memory storing program instructions. The program instructions, when executed by the one or more processors, may cause the one or more processors to generate, in response to receiving at least a first selection of a first image segment of a first image, a first extracted image segment including at least pixel data corresponding to pixels of the first image segment and metadata indicative of at least one of the first image or a source location of the first image. The program instructions may further cause the one or more processors to present at least the first extracted image segment as part of a collage and to receive a second selection of a second extracted image segment, the second extracted image segment corresponding to a second image, and to present the second extracted image segment together with the first extracted image segment as part of the collage, the first extracted image segment and the second extracted image segment being independently transformable when presented as part of the collage.

[0087] Optionally, the program instructions, when executed by the one or more processors, may further cause the one or more processors to present at least a plurality of extracted image segments, wherein the second extracted image segment is included within the plurality of extracted image segments and is selected from the plurality of extracted image segments. Optionally, at least one of the plurality of extracted image segments may be at least one of a popular extracted image segment, a trending extracted image segment, an extracted image segment that is visually similar to and related to the first extracted image segment and that allows for purchase of an object depicted in the extracted image segment, or an extracted image segment generated by a user who selected the first extracted image segment. Optionally, the program instructions, when executed by the one or more processors, may further cause the one or more processors to at least determine a seller of the object represented in the first extracted image segment, associate the seller with the first extracted image segment and present the seller with the first extracted image segment and concurrently with the presentation of the first extracted image segment and the second extracted image segment as part of the collage, and a purchasable indication indicates that the object represented in the first extracted image segment is purchasable by interaction with the first extracted image segment. Optionally, the program instructions, when executed by the one or more processors, may further cause the one or more processors to at least receive a request to transform the collage, determine that the request is from a second user different from the first user that created the collage, and in response to determining that the request is from the second user, generate a duplicate collage from the collage, maintain as part of the metadata of the duplicate collage a link or other identifier indicating that the duplicate collage was generated based on the collage, and transform the duplicate collage in response to the request.Optionally, the request may be at least one of a request to add a third extracted image segment to the collage, remove a second extracted image segment from the collage, remix a presentation of the first extracted image segment in the collage with the second extracted image segment in the collage, transform a position of the first extracted image segment, or transform a size of the first extracted image segment. Optionally, the program instructions, when executed by the one or more processors, may further cause the one or more processors to at least receive a request to transform the collage, determine that the request is from a user who created the collage or that the collage is a collaborative collage, and, in response to determining that the request is from a user or that the collage is a collaborative collage, transform the collage in response to the request. Optionally, the program instructions, when executed by the one or more processors, may further cause the one or more processors to at least receive a modification to the first image segment, modify which pixels of the first image are included in the first image segment, in response to the modification, generate a modified image segment based at least in part on the modification, and update training of a deep neural network ("DNN") based at least in part on the modified image segment, the DNN being trained to generate the one or more image segments from the input image. Optionally, the program instructions, when executed by the one or more processors, may further cause the one or more processors, in response to the modification, to reprocess the first image segment to determine objects represented in the first image segment.

[0088] Implementations disclosed herein include processing the first image with a DNN to determine a first image segment corresponding to an object depicted in the first image, where the first image segment is less than all of the first image; presenting the first image segment and the first image on a display such that the first image segment is visually distinct from the first image; receiving a modification of the first image segment to modify which pixels of the first image are included in the first image segment in response to the modification; generating an updated first image segment; and selecting the updated first image segment in response to the selection of the updated first image segment. and presenting the first extracted image segment as part of a collage including at least a second extracted image segment representing a second object extracted from the second image, where the first extracted image segment is adjustable relative to the second extracted image segment, where the first extracted image segment is adjustable relative to a size of the first extracted image segment when presented on the collage, and where the first extracted image segment is adjustable relative to a position of the first extracted image segment when presented on the collage. Optionally, the method may further include updating the training of the DNN based at least in part on the first image and the updated first image segment. Optionally, the display may be included on a user device and / or the DNN may operate on the user device. Optionally, the first extracted image segment may include metadata, which may include one or more of: an indication of the first image, an indication of a source location of the first image, a link to a website from which the object can be purchased, additional information about the object, reviews of the object, a link to a second collage that includes the first extracted image segment, the popularity of the first extracted image segment, or an indication of a user who created the first extracted image segment.Optionally, the method may further include presenting as part of the collage an indication of the total number of extracted image segments included in the collage.

[0089] The above aspects of the present disclosure are meant to be illustrative. They are selected to illustrate the principles and applications of the present disclosure, and are not intended to be exhaustive or to limit the present disclosure. Many modifications and variations of the aspects of the present disclosure may be apparent to those skilled in the art. Those skilled in the art with ordinary skill in the fields of computers, communications, media files, and machine learning should recognize that the components and process steps described herein may be compatible with other components or steps, or combinations of components or steps, and still achieve the advantages and benefits of the present disclosure. Moreover, it will be apparent to those skilled in the art that the present disclosure may be practiced without some or all of the specific details and steps disclosed herein.

[0090] Furthermore, with respect to one or more methods or processes of the present disclosure shown or described herein, including but not limited to the flow charts depicted in Figures 2-6 and 8, the order in which such methods or processes are presented is not intended to be construed as any limitation on the scope of the claims, and any number of steps or boxes of a method or process described herein can be combined in any order and / or in parallel to carry out a method or process described herein. Furthermore, some process steps or boxes may be optional. Also, the drawings herein are not drawn to scale.

[0091] Aspects of the systems of the present disclosure may be implemented as computer methods or articles of manufacture, such as memory devices or non-transitory computer-readable storage media. The computer-readable storage media may be readable by a computer and may include instructions for causing a computer or other device to perform the processes described in the present disclosure. The computer-readable storage media may be implemented by volatile computer memory, non-volatile computer memory, hard drives, solid-state memory, flash drives, removable disks, and / or other media. Additionally, one or more components of the modules and engines may be implemented in firmware or hardware.

[0092] Unless otherwise noted, disjunctive language, such as phrases such as "at least one of X, Y, or Z" or "at least one of X, Y, and Z," is understood in the context in which it is commonly used to state that an item, term, etc. can be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is not generally intended, and should not imply, that an implementation requires at least one of X, Y, or Z in order for each to be present.

[0093] Unless expressly stated otherwise, articles such as "a" or "an" should generally be construed to include one or more of the listed items. Thus, phrases such as "an apparatus configured to" or "an apparatus operable to" are intended to include one or more of the listed apparatus. Such one or more listed apparatuses may also be collectively configured to perform the listed items. For example, "a processor configured to perform items A, B, and C" may include a first processor configured to perform item A in conjunction with a second processor configured to perform items B and C.

[0094] As used herein, terms of degree used herein, such as "about," "approximately," "generally," "near," or "substantially," refer to a value, amount, or characteristic that is close to a stated value, amount, or characteristic that still performs a desired function or achieves a desired result. For example, terms such as "about," "approximately," "generally," "near," or "substantially" can refer to an amount that is less than 10%, less than 5%, less than 1%, less than 0.1%, and less than 0.01% of the stated amount.

[0095] In particular, unless otherwise stated or understood otherwise within the context in which it is used, conditional language such as, for example, "can," "could," "might," or "may" is generally intended to convey in an open-ended manner that a particular implementation can or may include certain features, elements, and / or steps, but is not constrained or required to include them. Similarly, terms such as "include," "including," and "includes" are generally intended to mean "including, but not limited to." Thus, such conditional language is generally not intended to imply that features, elements, and / or steps are in any way required by one or more implementations, or that one or more implementations necessarily include logic for determining whether those features, elements, and / or steps should be included or performed in any particular implementation, with or without user input or prompting.

[0096] Although the present invention has been described and illustrated with respect to exemplary implementations thereof, the foregoing and various other additions and omissions can be made therein and thereto without departing from the spirit and scope of the disclosure.

Claims

1. 1. A computer-implemented method, comprising: processing a first image to determine a first image segment corresponding to less than all of the first image and corresponding to an object depicted in the first image; presenting, on a user device, the first image and the first image segment such that the first image segment is visually distinct from the first image; In response to selection of the first image segment, at least pixel data corresponding to pixels of the first image segment; forming a first extracted image segment comprising metadata indicative of at least one of the first image or a source location of the first image; presenting the first extracted image segment as part of a collage including at least a second extracted image segment representing a second object extracted from a second image; The first extracted image segment may be transformed relative to the second extracted image segment; the first extracted image segment may be transformed relative to a size of the first extracted image segment when presented on the collage; A computer-implemented method, wherein the first extracted image segment may be transformed relative to a position of the first extracted image segment when presented on the collage.

2. The computer-implemented method of claim 1 , further comprising receiving a modification of the first image segment to modify which pixels of the first image are included in the first image segment.

3. receiving an indication of a region of interest within the first image; The computer-implemented method of claim 1 , wherein processing comprises processing a portion of the first image within the region of interest to determine the first image segment.

4. The computer-implemented method of claim 1 , 2 or 3 , wherein processing the first image is performed on the user device.

5. 1. A computing system comprising: one or more processors; a memory storing program instructions that, when executed by the one or more processors, cause the one or more processors to at least: In response to receiving a first selection of a first image segment of a first image, at least pixel data corresponding to pixels of the first image segment; forming a first extracted image segment comprising metadata indicative of at least one of the first image or a source location of the first image; presenting the first extracted image segment as part of a collage; receiving a second selection of a second extracted image segment, the second extracted image segment corresponding to a second image; and presenting the first extracted image segment with the second extracted image segment as part of the collage; A computing system, wherein the first extracted image segment and the second extracted image segment are independently transformable when presented as part of the collage.

6. The program instructions, when executed by the one or more processors, cause the one or more processors to at least:

6. The computing system of claim 5, further comprising: presenting a plurality of extracted image segments, the second extracted image segment being included within the plurality of extracted image segments and selected from the plurality of extracted image segments.

7. 7. The computing system of claim 6, wherein at least one of the plurality of extracted image segments is at least one of a popular extracted image segment, a trending extracted image segment, an extracted image segment that is visually similar to and related to the first extracted image segment and that allows for purchase of an object depicted in the extracted image segment, or an extracted image segment generated by a user who selected the first extracted image segment.

8. The program instructions, when executed by the one or more processors, cause the one or more processors to at least: determining a seller of an object represented in the first extracted image segment; associating the merchant with the first extracted image segment; 8. The computing system of claim 5, 6, or 7, further comprising: presenting, together with the first extracted image segment and simultaneously with the presentation of the first extracted image segment and the second extracted image segment as part of the collage, a purchasable indication indicating that the object depicted in the first extracted image segment is purchasable through interaction with the first extracted image segment.

9. The program instructions, when executed by the one or more processors, cause the one or more processors to at least: receiving a request to transform the collage; determining that the request is from a second user different from a first user who created the collage; in response to determining that the request is from the second user; generating a replica collage from the collage; maintaining as part of the metadata of the reproduction collage a link or other identifier indicating that the reproduction collage was generated based on the collage; 9. The computing system of claim 5, 6, 7, or 8, further configured to: in response to said request, transform said replica collage.

10. The request, adding a third extracted image segment to the collage; and removing the second extracted image segment from the collage; and remixing a representation of the first extracted image segment and the second extracted image segment of the collage; Transforming the position of the first extracted image segment; or and converting a size of the first extracted image segment.

11. The program instructions, when executed by the one or more processors, cause the one or more processors to at least: receiving a request to transform the collage; determining that the request is from a user who created the collage or that the collage is a collaborative collage; in response to determining that the request is from the user or that the collage is a collaborative collage; 11. The computing system of claim 5, 6, 7, 8, 9, or 10, further configured to: transform the collage in response to the request.

12. The program instructions, when executed by the one or more processors, cause the one or more processors to at least: receiving a modification of the first image segment to modify which pixels of the first image are included in the first image segment; In response to said amendment, generating a modified image segment based at least in part on the modification; and 12. The computing system of claim 5, 6, 7, 8, 9, 10, or 11, further comprising: updating training of a deep neural network ("DNN") based at least in part on the modified image segments, the DNN being trained to generate one or more image segments from an input image.

13. 1. A method comprising: processing a first image with a deep neural network ("DNN") to determine a first image segment corresponding to an object depicted in the first image, the first image segment being less than all of the first image; presenting, on a display, the first image segment and the first image such that the first image segment is visually distinct from the first image; in response to receiving the modification of the first image segment, modifying which pixels of the first image are included in the first image segment to generate an updated first image segment; forming a first extracted image segment in response to selecting the updated first image segment, the first extracted image segment including pixel data corresponding to pixels of the updated first image segment; presenting the first extracted image segment as part of a collage including at least a second extracted image segment representing a second object extracted from a second image; the first extracted image segment is adjustable relative to the second extracted image segment; the first extracted image segment is adjustable with respect to a size of the first extracted image segment when presented on the collage; The method, wherein the first extracted image segment is adjustable relative to a position of the first extracted image segment when presented on the collage.

14. The method of claim 13 , further comprising updating training of the DNN based at least in part on the first image and the updated first image segment.

15. 15. The method of claim 13 or 14, further comprising presenting as part of the collage an indication of a total number of extracted image segments included in the collage.

Images