Method and apparatus for identifying and matching objects depicted in images

Abstract

The present disclosure relates to methods and apparatus for recognizing and matching objects depicted in images. The present disclosure provides a method comprising assigning an object category to a real-world object displayed in an image associated with a first user based on object recognition; and providing a notification to the first user that a content item provided by a second user has been associated with the object category; wherein the second user marks the content item as a real-world object displayed in an image associated with the second user, and wherein the content item is different from the image associated with the second user. A computer-readable storage medium stores one or more computer programs, and an apparatus includes a processor-based device.

Description

[0001] This application is a divisional application of the invention patent application with application number 201580051213.X, application date September 18, 2015, and invention name “Method and device for identifying and matching objects depicted in images”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application is a continuation of and claims the benefit of U.S. Patent Application No. 14 / 565,204, filed on December 9, 2014, entitled “METHOD AND APPARATUS FOR RECOGNITION AND MATCHING OF OBJECTS DEPICTED IN IMAGES,” which claims the benefit of U.S. Provisional Patent Application No. 62 / 057,223, filed on September 29, 2014, entitled “METHOD AND APPARATUS FOR RECOGNITION AND MATCHING OF OBJECTS DEPICTED IN IMAGES,” and also claims the benefit of U.S. Provisional Patent Application No. 62 / 057,223, filed on September 29, 2014, entitled “SCHEMES FOR RETRIEVING AND ASSOCIATING CONTENT ITEMS WITH REAL-WORLDOBJECTS USING AUGMENTED REALITY AND OBJECT RECOGNITION”, the entire contents and disclosure of which are hereby incorporated by reference herein in their entirety.

[0004] This application is related to U.S. patent application No. 14 / 565,236, filed on December 9, 2014, entitled “SCHEMES FOR RETRIEVING AND ASSOCIATING CONTENT ITEMS WITH REAL-WORLD OBJECTS USING AUGMENTED REALITY AND OBJECT RECOGNITION,” the entire contents and disclosure of which are hereby fully incorporated herein by reference. Background of the Invention

[0005] 1. Field of the Invention

[0006] The present invention relates generally to computer software applications, and more particularly to object recognition and matching devices and software.

[0007] 2. Related technical discussions

[0008] The sharing of user-generated content has developed into one of the most popular features on the Internet. In contrast to Web sites where people are limited to passively viewing content, so-called "Web 2.0" sites allow users, as creators of user-generated content, to interact and collaborate with each other in virtual communities in social media conversations. SUMMARY OF THE INVENTION

[0009] One embodiment provides a method comprising: identifying one or more objects in one or more real-world scene images associated with a user; adding the identified one or more objects to a real-world object list associated with the user; assigning an object category to each object in the real-world object list based on object recognition; and providing a notification to the user that a content item has been associated with the object category assigned to one of the objects on the real-world object list associated with the user.

[0010] Another embodiment provides an apparatus comprising: a processor-based device; and a non-transitory storage medium storing a set of computer-readable instructions configured to cause the processor-based device to perform multiple steps, the steps comprising: identifying one or more objects in one or more real-world scene images associated with a user; adding the identified one or more objects to a real-world object list associated with the user; assigning an object category to each object in the real-world object list based on object recognition; and providing a notification to the user that a content item has been associated with an object category assigned to one of the objects on the real-world object list associated with the user.

[0011] Another embodiment provides a method comprising: identifying one or more objects in one or more images associated with a first user, wherein the identification is performed by an object recognition technique executed by a processor-based device; for each identified object, searching through one or more images associated with other users to identify one or more images of interest, wherein the one or more images of interest each depict an object that (i) has an appearance that substantially matches the identified object and (ii) has been specified by one of the other users to be included in the search; generating a notification based at least in part on any identified image of interest; and providing the notification to the first user.

[0012] Another embodiment provides a method comprising: identifying one or more objects in one or more real-world scene images associated with a first user; adding the identified one or more objects to a real-world object list associated with the first user; assigning an object category to each object in the real-world object list based on object recognition; receiving content items for the real-world objects in the real-world object list from the first user; and storing the content items and the object categories of the real-world objects on a server so that the content items can be retrieved by a second user viewing the objects in the object category of the real-world objects.

[0013] Another embodiment provides an apparatus comprising: a processor-based device; and a non-transitory storage medium storing a set of computer-readable instructions configured to cause the processor-based device to perform multiple steps, the steps comprising: identifying one or more objects in one or more real-world scene images associated with a first user; adding the identified one or more objects to a real-world object list associated with the first user; assigning an object category to each object in the real-world object list based on object recognition; receiving content items for real-world objects in the real-world object list from the first user; and storing the content items and the object categories of the real-world objects on a server so that the content items can be retrieved by a second user viewing an object in the object category of the real-world object.

[0014] A better understanding of the features and advantages of various embodiments of the present invention will be obtained by reference to the following detailed description and accompanying drawings that set forth illustrative embodiments in which the principles of embodiments of the invention are utilized. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other aspects, features and advantages of embodiments of the present invention will become more apparent from the following more particular description of the present invention presented in conjunction with the following drawings, in which:

[0016] Figure 1 , Figure 2 and Figure 3 is a flow chart illustrating a method of object recognition and matching according to some embodiments of the present invention;

[0017] Figure 4 is a diagram illustrating recognition of objects in an image of a real-world scene according to some embodiments of the present invention;

[0018] Figure 5 , Figure 6 , Figure 7 and Figure 8 is a diagram illustrating a user interface for object-based content sharing according to some embodiments of the present invention;

[0019] Fig. 9 is a block diagram illustrating a system that can be used to perform, implement, and / or execute any of the methods and techniques shown and described herein, according to some embodiments of the present invention. Detailed Description

[0020] Some embodiments of the present invention provide methods and devices for what in some embodiments may be described as object-based content sharing. Object-based content sharing provides a novel way to share stories, memories, and recorded media based on real-world objects. Using images associated with a user, the system can automatically identify various objects in the user's environment in some embodiments. The system then provides a user interface for the user to publish and view content based on these identified objects.

[0021] Under object-based content sharing, users can interact with each other based on real-world objects they own and / or encounter. For example, users who have a collection of comic books but live miles apart can connect to each other when the system recognizes the same comic book in an image associated with the users. Users can then share user-generated content and discuss their common interests through the object-based content service.

[0022] In some embodiments, the system can further provide configuration flexibility for how objects are matched and how objects are grouped for sharing to let users take charge of their privacy and their social networking experience. In some embodiments, the object-based content sharing system allows users to connect with other users who share common interests, hobbies, styles, tastes, lifestyles, etc. with the user based on objects they own and / or encounter.

[0023] refer to Figure 1 , shows an example of a method 100 for providing a notification of a content item associated with an object according to some embodiments of the present invention. In some embodiments, the steps of method 100 may be performed by one or more server devices, user devices, or a combination of server and user devices.

[0024] In step 110, the system identifies objects in an image of a real-world scene associated with the user. The image of the real-world scene may include one or more of the following: photos or videos in an online album associated with the user's online profile; local storage on the user's device; cloud storage space associated with the user; and a collection of images that the user has "tagged" in a social media service, etc. The image may be captured at multiple locations and over multiple time periods, such as hours, days, months, or years. In some embodiments, objects in images viewed by the camera may also be identified when the camera is panning, where the image is not permanently stored on the device. The object may be an object in the background of a photo of the user and / or other people. The system may run an image or object recognition algorithm on an image associated with the user and identify one or more objects in the image. In some embodiments, the system may only identify inanimate objects in images associated with the user.

[0025] In general, the system can utilize various known computer object recognition techniques to recognize objects in images of real-world scenes. In some embodiments, object recognition can use an appearance-based method that compares an image with a reference image of a known object to recognize the object. Examples of appearance-based methods include edge matching, grayscale matching, receptive field response histograms, and the like. In some embodiments, object recognition can use a feature-based method that relies on matching object features and image features. Examples of feature-based methods include morphological clustering, geometric hashing, scale-invariant feature transforms, interpretation trees, and the like. The system can use one or more object recognition methods in combination to improve the accuracy of object recognition.

[0026] In step 120, the real-world object identified in step 110 is added to a list of objects associated with the user. The list of objects may be maintained on a content sharing server and associated with the user's user profile. In some embodiments, an image of the object is stored on the list in step 120. In some embodiments, attributes of the identified object are stored to the list in step 120. For example, if an object is identified as 3 Slim, then describe " 3 Slim" and attributes such as "game console" and "red" can be added to the list. In some embodiments, the list stores one or more images and attributes of the object. In some embodiments, before adding an object to the list, the system first checks for redundancy. For example, the system may first check to see if an object matching the object in the image is already on the list. In some embodiments, the system provides a user interface for a user to access and edit the object list. For example, a user can use the user interface to add objects to and remove objects from a list of objects associated with him / her. In some embodiments, the user can also edit and / or correct attributes associated with an object. For example, a user may be able to select an object on the list. 3 Slim, and enter the model and / or serial number. In another example, the system can extract the object in the image if it is unable to match the object with a known object. The user can manually enter the attribute information of the object.

[0027] In step 130, an object class is assigned to one of the multiple objects on the object list. An object class may refer to a unique object, a group of visually similar objects, or a group of objects that share one or more common attributes. In some embodiments, an object class is assigned to an image based on a comparison of an image of an object with other images of an object to which the object class has been assigned. In some embodiments, the assignment in step 130 is based on one or more attributes of an object identified by object recognition. An object class may be assigned to a real-world object based on running an image recognition algorithm for one or more images of an object in a real-world scene. In some embodiments, features and / or attributes of the identified real-world object are compared and matched with attributes associated with the object class and / or reference images of known objects. Features and attributes may include visual features, such as color, shape, outline, etc. In some embodiments, features and attributes may include attributes determined by image recognition. For example, a system may compare an image with a database of known objects to determine additional information related to the image, such as image type, name, and other attributes.

[0028] For example, an object may be identified as red based on its color, shape, brand image, etc. 3 ultra-lightweight console. The system can then be red 3. The Ultra-Lightweight Console assigns one or more object categories, such as: Console, 3 consoles, 3 ultra-light consoles and red 3 ultra-lightweight console, etc. The object category assignment may be based on matching the properties of the real-world object with the properties associated with each of these object categories. In some embodiments, two or more object categories may be assigned to an object based on the properties of the object. In some embodiments, the object categories may have a hierarchical relationship to each other and are presented in that relationship for user configuration. In some embodiments, a user may be allowed to edit the object category assigned to each object on his / her object list. For example, if a user does not want to receive a request for an object marked as " 3 Console" object category, he / she can place his / her console in the 3 Console" object class and assign only the "Red 3 Ultra-Lightweight Console" object categories. In some embodiments, a list of suggested object categories can be determined using object recognition and provided to the user for selection by the user. In some embodiments, the user can manually search for object categories on the system to associate with objects on his / her object list.

[0029] In step 140, the system provides a notification that the content item has been added to the object category that has been assigned to the object on the user's object list. (It was also assigned the title “Red 3 Ultra-Lightweight Console" object class), the system can provide the user with a red 3 Ultra-lightweight consoles exist on his / her object list. The notification can be one or more of the following: pop-up notification, sound notification, vibration, notification text or icon in the content sharing user interface based on the object, notification message in the social media service, text message and / or mail message, etc. For example, once a new content item is added, the system can make the user's device emit a sound or vibrate. In another example, when the user opens the content sharing user interface or social network service based on the object, the system can display an icon or message to the user to indicate that the new content item is available for viewing. In some embodiments, the user can configure what type of notification to receive. In some embodiments, the user can individually configure the type of notification to be received for each object and / or object category associated with the user. In some embodiments, the user can individually configure whether to receive notification for each object and / or object category associated with the user.

[0030] In some embodiments, before step 140, the system first determines whether the content item should be shared with the first user based on the sharing settings of the content item and / or the preference settings of the first user. In some embodiments, the author of the content item can limit the sharing of the content item. For example, the author may only allow users who are connected to it and / or pay attention to it in a social network service to watch the content. In some embodiments, the author can limit who can watch the content item based on age, geographic location, time, etc. For example, the author may wish to mark the content only for viewing in the next few years, or only for viewing in a time period. In some embodiments, the first user can also close the notification based on the author of the content item. For example, the first user may wish to receive notifications to the content item only from his friends, the people he pays attention to, the people in his social group, etc. In another example, the user may be able to selectively shield the content item from the selected author. In these cases, the system can only proceed to step 140: determine whether the content item should be shared with the first user.

[0031] refer to Figure 2 , shows an example of a method 200 for providing a notification of an image of interest according to some embodiments of the present invention. In some embodiments, the steps of method 200 may be performed by one or more server devices, user devices, or a combination of server and user devices.

[0032] In step 210, real-world objects in an image of a real-world scene associated with a user are identified. The image of the real-world scene may include one or more of the following: photos or videos in an online album associated with the user's online profile; local storage on the user's device; cloud storage space associated with the user; and a collection of images that the user has "tagged" in a social media service, etc. The image may be captured at multiple locations and over multiple time periods, such as hours, days, months, or years. In some embodiments, objects in images viewed by a camera may also be identified when the camera is panning, where the image is not permanently stored on the device. The object may be an object in the background of a photo of the user and / or other people. The system may run an image or object recognition algorithm on an image associated with the user and identify one or more objects in the image. In some embodiments, the system may only identify inanimate objects in images associated with the user.

[0033] In step 220, for each image identified in step 210, the system searches through one or more images associated with other users to identify one or more images of interest. The images of interest may depict an object that substantially matches the identified object in appearance and has been specified by one of the other users for inclusion in the search.

[0034] In some embodiments, images of the identified real-world objects are compared to images of objects associated with other users in order to match objects with substantially matching appearances. In some embodiments, features and / or attributes of the identified real-world objects are compared and matched to features and / or attributes of objects associated with other users. Features and attributes may include visual features, such as color, shape, outline, etc. In some embodiments, features and attributes may include attributes determined by object recognition. For example, the system may compare the image and / or visual attributes of the real-world object to a database of known objects to determine additional information about the image, such as image type, name, and other attributes.

[0035] The system can further determine whether each image of interest has been specified to be included in the search by one of the other users based on the sharing settings of each image of interest. In some embodiments, the sharing settings of the image of interest can be configured by the user associated with the image of interest. The sharing settings can limit who can view the content items associated with the image of interest. For example, the user can limit the sharing of content based on whether the user is connected through a social networking service. The image of interest can only be included in the search of the user's friends, followers, group members, etc. In some embodiments, the user can also limit sharing based on the age, location, etc. of the user associated with the identified object. In some embodiments, the user can limit the time period during which the image of interest can be searched. For example, the user can configure only two days to search for the image of interest.

[0036] In step 230, when an image of interest is found in step 220, a notification is generated. For example, if the image of a baseball cap has an appearance that substantially matches the appearance of the baseball cap identified in step 210 and has been specified by other users to be included in the search, a notification is generated in step 230. The notification can be one or more of a pop-up notification, a sound notification, a vibration, a notification text or icon in an object-based content sharing user interface, a notification message in a social media service, a text message, and / or an email message, etc. In some embodiments, the notification can include a list of the identified images of interest. In some embodiments, whether and how to provide a notification in step 230 is further based on the preferences configured by the first user.

[0037] In step 240, the notification generated in step 230 is provided to the user. In some embodiments, once an image of interest is found, the system can cause the user's device to emit a sound or vibrate. In some embodiments, when the user opens an object-based content sharing user interface or a social networking service, the system displays an icon or message to the user to indicate that one or more matching images of interest have been found. In some embodiments, the user can configure what type of notification to receive.

[0038] In some embodiments, the system can further determine whether the content item has been associated with an object whose appearance is substantially matched with the identified object. A user can associate a content item with an object through an object-based content sharing system. Users with substantially matching objects can share their memories and experiences with each other by associating content items such as text notes, images, audio clips, video clips, and hyperlinks with objects. The notification in step 240 can also include an indication that any content item has been associated with or marked to any object whose appearance is substantially matched with any identified object.

[0039] refer to Figure 3 , shows an example of a method 300 for associating a content item with a real-world object according to some embodiments of the present invention. In some embodiments, the steps of method 300 may be performed by one or more server devices, user devices, or a combination of server and user devices.

[0040] In step 310, the system identifies objects in a real-world scene associated with the user. The image of the real-world scene may include one or more of the following: photos or videos in an online album associated with the user's online profile; local storage on the user's device; cloud storage space associated with the user; and a collection of images that the user has "tagged" in a social media service, etc. The image may be captured at multiple locations and over multiple time periods, such as hours, days, months, or years. In some embodiments, objects in images viewed by the camera may also be identified when the camera is panning, where the image is not permanently stored on the device. The object may be an object in the background of a photo of the user and / or other people. The system may run an image or object recognition algorithm on an image associated with the user and identify one or more objects in the image. In some embodiments, the system may only identify inanimate objects in images associated with the user.

[0041] In step 320, the real-world object identified in step 110 is added to a list of objects associated with the user. The list of objects may be maintained on a content sharing server and associated with the user's user profile. In some embodiments, an image of the object is stored to the list in step 320. In some embodiments, attributes of the identified object are stored to the list in step 320. For example, if an object is identified as 3 Slim, then describe " 3 Slim" and attributes such as "game console" and "red" can be added to the list. In some embodiments, the list stores one or more images and attributes of the object. In some embodiments, before adding an object to the list, the system first checks for redundancy. For example, the system may first check to see if an object matching the object in the image is already on the list. In some embodiments, the system provides a user interface for a user to access and edit the object list. For example, a user can use the user interface to add objects to and remove objects from a list of objects associated with him / her. In some embodiments, the user can also edit and / or correct attributes associated with an object. For example, a user may be able to select an object on the list. 3 Slim, and enter the model and / or serial number. In another example, the system can extract the object in the image if it is unable to match the object with a known object. The user can manually enter the attribute information of the object.

[0042] In step 330, an object class is assigned to one of the plurality of objects on the object list. An object class may refer to a unique object, a group of visually similar objects, or a group of objects that share one or more common attributes. The object class may be assigned to a real-world object based on running an image recognition algorithm on one or more images of the object in a real-world scene. In some embodiments, features and / or attributes of the identified real-world object are compared and matched to attributes associated with each object class. For example, an object identified as being red may be assigned an object class. 3 Ultra-lightweight console objects can be of all object categories 3. 3Ultra-light and red 3 ultra-lightweight console, etc. The features and attributes may include visual features, such as color, shape, outline, etc. In some embodiments, the features and attributes may include attributes determined by image recognition. For example, the system may compare the image to a database of known objects to determine additional information about the image, such as image type, name, and other attributes. In some embodiments, if the object identified in step 310 does not match any existing object categories, the user may choose to create a new object category by defining common attributes of the object categories.

[0043] In some embodiments, the assignment in step 330 is based on one or more attributes of the object identified by object recognition. In some embodiments, two or more object categories may be assigned to an object based on its attributes. In some embodiments, a user may be allowed to edit the object category assigned to each object on his / her object list. For example, if a user does not want to be associated with all 3. If the owner of the console shares the content item, he / she can share his / her console in 3. De-assignment in object categories. In some embodiments, a list of suggested object categories can be determined using object recognition and provided to the user for selection by the user. In some embodiments, the user can manually search for object categories on the system to associate them with objects on his / her object list. Generally speaking, in some embodiments, steps 310, 320, and 330 can be similar to steps 110, 120, and 130, respectively. In some embodiments, steps 310, 320, and 330 are the same as steps 110, 120, and 130.

[0044] In step 340, a content item for marking to a real-world object is received. The content item may be one or more of a video, audio, image, and text annotation. In some embodiments, the user interface may include an option for capturing one or more of a video clip, an audio clip, and an image for marking to an object. In some embodiments, the user may further configure the sharing settings of the content item in step 340. For example, the user may limit the sharing of the content item so that the content item is only available for viewing by other users connected through a social networking service. In some embodiments, the sharing settings of the content item may further limit sharing based on age and / or geographic location. Geographic location restrictions may be used by users to distinguish two visually similar items from each other. For example, a user may use the GPS location of a park bench to limit the sharing of the content item marked to the park bench so that the content item will only match the exact park bench in other users' photos. The location restriction may be location metadata information based on the current location of the user device and / or an image of a real-world scene containing an object. In some embodiments, the location information may be one of the attributes defined for an object class, rather than part of the sharing settings. For example, a bench at location A and a nearly identical-looking bench at location B may be in two separate object categories. In some embodiments, a user may be able to select one or more object categories to associate with a content item.

[0045] In step 350, the content item and the associated object category of the real-world object are stored so that the content item can be retrieved by another user whose object on his object list has been assigned the same object category. For example, the content item can be stored by the attributes of the real-world object and / or the assigned object category. When another user has an object with a shared selected common attribute on his object list, the content item can be provided to the other user for viewing. In some embodiments, the availability of the content item can be further based on sharing settings associated with the content item, such as social network connections and location restrictions.

[0046] refer to Figure 4 , there is an example diagram illustrating the construction of a list of objects associated with a user according to some embodiments of the present invention. Figure 4In the example of FIG. 4 , images of real-world scenes 410 and 420 associated with a user are processed by a server 440 to generate a list of objects associated with the user as shown in a user interface 430. The images of real-world scenes 410 and 420 may include photos or videos in online albums, images stored on the user's device, images in albums on cloud storage, and images "tagged" by the user in social media services, etc. The images may be a collection of images captured over a period of hours, days, months, or years and captured at different locations. In some embodiments, objects in images viewed by a camera may also be identified when the camera is panning, where the images are not permanently stored on the device. The system may run an object recognition algorithm on the images associated with the user and identify one or more objects in the images. In some embodiments, the system only identifies inanimate objects in images associated with the user.

[0047] like Figure 4 As shown, the system identifies a soldier doll 412 and an alarm clock 413 in a first image of a real-world scene 410, and identifies a trophy 421 and a baseball cap 422 in a second image of a real-world scene 420. Each of the identified objects is then added to a list of objects associated with the user shown in an object list user interface 430. The user can use the object list user interface 430 to associate content items with the identified objects. The user can further configure the object category and sharing settings of the object through the object list user interface 430.

[0048] exist Figure 4 , a user has added an audio clip to trophy list 433 and configured the object category as "Joe Smith's 1985 baseball trophy". The sharing setting for trophy list 433 has been set to "Everyone" without location restrictions. Under this setting, anyone who has an image of Joe Smith's 1985 baseball trophy in an image associated with it can listen to the audio clip tagged to the trophy. For example, someone who wins this trophy in the next few years will be able to listen to the audio clip left by the user who reposted his / her memory of the trophy.

[0049] Also in Figure 4 , the user has tagged an image to a baseball cap list 434. The object category for baseball cap 434 is "Dodgers baseball cap" and the sharing settings are set to "Friends only" and "Within 5 miles of San Diego, California." Under this setting, friends of the user on the social network who have images of Dodgers baseball caps within 5 miles of San Diego, California may be able to view image 436 of the user wearing the baseball cap.

[0050] The user has not associated any content items with the soldier doll list 431 and the alarm list 432. The user can select one of the "Add Notes", "Add Image", "Add Video", and "Add Audio" buttons to associate new content with the object.

[0051] Although sharing and location settings are shown as settings associated with the identified object, in some embodiments, one or more of the sharing and location settings may be associated with a content item instead. A user may associate multiple content items with an object. Different content items associated with the same object on the object list may have different object categories, sharing settings, and / or location settings.

[0052] refer to Figure 5 , shows an example of a user interface 500 for viewing content items associated with a real-world object in an object list according to some embodiments of the present invention. When a user logs into an object-based content sharing system, the user can choose to view a list of objects associated with him / her and view any content items tagged to or associated with the object by other users. In some embodiments, each object list 511, 512, 513, and 515 includes an image of an object 530, any content items tagged to the object 520, and an annotation option 540 for adding additional content items to the object.

[0053] The image of the object 530 may be an image extracted from an image of a real-world scene associated with the user and / or an image of an object representing an assigned object category of the object. The associated content item 520 may include one or more of: an avatar associated with the user adding the content item, the content item, a preview of the content item, and an option to play the content item. Annotation options 540 allow a user to create and / or attach new content items and to attach content to the user. Figure 4 Configure the sharing settings in a similar manner as shown in .

[0054] In some embodiments, user interface 430 and user interface 500 may be combined so that a user can view content items on an object list and add content items to the object list in the same user interface. In some embodiments, object lists 511, 512, 513, and 515 may be labeled in the order of the latest content items for each object and / or sorted in the order of the latest objects associated with the user.

[0055] refer to Figure 6, another example of a user interface 600 for object-based content sharing is shown. The user interface 600 displays a content sharing page for an object or object category. The page may include one or more images 610 of the associated object, an option 620 for adding content, and content items 631, 632, and 633 tagged to the object or object category.

[0056] exist Figure 6 , the page of "San Diego Padres Blue Logo Hat" is shown. Each of content items 631, 632 and 633 may include one or more of the following: an avatar associated with the user who adds the content item, a content item, a preview of the content item and an option for playing the content item. In some embodiments, user interface 600 can only be viewed by such a user as follows: the object on its object list matches the hat shown in image 610 and / or has the common attribute associated with "San Diego Padres Blue Logo Hat" object category. For example, if a user has taken a photo wearing a priest blue logo hat, the system can identify the hat as belonging to this object category so that the hat is distinguished and the user is connected to the object category shown.

[0057] In some embodiments, the content of an object page can only be viewed by users who have matching objects on their object lists. In some embodiments, user interface 600 can be viewed by everyone, but only users who have objects matching the object category on their lists can add content items to the page. In some embodiments, the administrator of an object page can choose to configure who can view the object page and / or add content items to the object page.

[0058] In some embodiments, the user will only see content items for which the sharing setting allows the user to view the content items. In some embodiments, the user interface is an alternative view of interfaces 430 and 500. In some embodiments, the object page can be accessed by selecting Figure 5 and Figure 6 One of the objects in the list and / or as referenced below Figure 7 and Figure 8 Describes the objects in the image of the real-world scene.

[0059] refer to Figure 7 and Figure 8, shows an additional example of a user interface for an object-based content sharing system according to some embodiments of the present invention. In user interface 710, a user can select an identified object 714 in a real-world scene 712 to tag a content item to the object. Real-world scene 712 can be a scene currently viewed by the device or an image previously captured by the device or another device. The user can view a photo or video stored on the user device or stored remotely and select one of the objects in the image of the real-world scene, such as a car 714, to tag.

[0060] After selecting the car 714, a tagging user interface 720 may be provided to the user for the user to enter or attach content items for the object. The tagging user interface 720 may include an image of the selected object 722 and one or more options 724 for attaching content items. The options may include "add notes", "add video", "add image", and "add audio". In some embodiments, the tagging user interface 720 may also include options for adding content items as described above. Figure 4 Describes options for configuring and assigning shared settings for object categories and content items.

[0061] User interface 810 shows a notification of content items tagged to an object in an image of a real-world scene 812. Real-world scene 812 may be a scene currently viewed by the device or an image previously captured by the device or another device. For example, a user may browse an album of photos or videos, and a notification may be provided if the image currently being viewed contains one or more tagged objects. A graphical indicator such as icon 814 may be displayed with the image of the real-world scene 812 to indicate to the user that the object in the image has been tagged with a content item available to the user.

[0062] When a graphical indicator such as icon 814 is selected in the image of the real-world scene, an associated content item 824 may be displayed in the content item viewing interface 820. In some embodiments, the content item may be overlaid onto the image of the real-world scene 822 as shown, or displayed in full screen.

[0063] Although Figure 8 The user interface in is shown as being displayed by a mobile device, but the user interface can be displayed on any type of user device with a display and can be part of an application ("app") and / or a website.

[0064] Fig. 9900 is a block diagram illustrating a system that can be used to perform, implement, and / or execute any of the methods and techniques shown and described herein according to an embodiment of the present invention. Object-based content sharing system 900 includes user devices 910, 920, and 930, a content sharing server 940, and a social network server 970 that communicates via a network 905.

[0065] The user device 910 may be any user device, such as a desktop computer, a laptop computer, a smartphone, a tablet computer, a tablet-like device, a head-mounted display, and / or a wearable device. The user device 910 may include a processor 911, a memory 912, a network interface 913, a display 915, and one or more other input / output devices 916. The processor 911 is configured to execute computer-readable instructions stored in the memory 912 to facilitate reference to the user device 910. Figure 1-3 One of the multiple steps of the method described. The memory 912 may include RAM and / or hard drive memory devices. The network interface 913 is configured to transmit data to at least the content sharing server 940 via the network 905 and receive data from it. The display 915 can be a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a liquid crystal on silicon (LCoS) display, an LED light emitting display, and the like. In some embodiments, the display 915 is a touch display configured to receive input from a user. In some embodiments, the display 915 is a head-mounted display that projects an image into a lens or the wearer's eyes. The display can be used to display the above referenced in different ways. Figure 4-8 The user device 910 may also include other input / output devices 916, such as a microphone, a speaker, a tilt sensor, a compass, a USB port, a camera, a graphics processor, etc. The input / output devices 916 may be used to enter and / or play back content items.

[0066] User devices 920 and 930 may be user devices operated by one or more other users to access object-based content sharing system 900. Fig. 9 Three user devices are shown in FIG. 9 , but any number of user devices and users may access system 900 .

[0067] Content sharing server 940 includes processor 942, memory 941 and network interface 944. In some embodiments, processor 942 is configured to execute computer readable instructions stored in memory 941 to perform the content sharing server 940. Figure 1-3 The content sharing server 940 may be connected to or include one or more of the object database 960 and the content item database 950 .

[0068] The content item database 950 may store information related to each of the following: tagged objects or object categories, content items tagged to each object or object category, and sharing settings associated with the content items and / or tagged objects. Figure 3 Step 350 in is performed by storing the information in a content item database 950 .

[0069] The object database 960 may include a database of known objects used by the system's object recognition algorithm to identify one or more real-world objects and their attributes. In some embodiments, an image of a real-world object is compared to one or more images of the object in the object database 960. When there is a match, the object database 960 may provide further information about the object, such as the object name, object type, object model, etc.

[0070] In some embodiments, one or more of the object database 960 and the content item database 950 can be part of the content sharing server 940. In some embodiments, the object database 960 and the content item database 950 can be implemented as a single database. In some embodiments, the content sharing server 940 also communicates with one or more of the object database 960 and the content item database 950 through the network 905. In some embodiments, the object database 960 can be maintained and controlled by a third party. In some embodiments, the object database 960 can be "learned" about new objects by receiving images and attributes provided by users, and adding the information provided by users to its database. For example, when a user takes a photo of a car and types its manufacturer and model, the object database may be able to discern another image of the same car and can provide information about the manufacturer and model of the same car. Although only one content sharing server 940 is shown, it should be understood that the content sharing server 940, the object database 960 and the content item database 950 can be implemented with one or more physical devices connected by a network.

[0071] The social network server 970 provides social network functions for users to connect with each other and build social networks and social groups. The social network server 970 can be a part of the content sharing server 940 or a third-party service. The connection and the built group in the social network service can be used to configure the sharing settings discussed in this article. For example, if the content item is configured to be shared only with the author's "friends", then the content sharing server 940 can query the social network status between the two users to the social network server 970 to determine whether the content item should be provided to the second user. In some embodiments, when the user configures the sharing settings, information can be retrieved from the social network server 979 so that the user can choose to share the content item among his / her friends and / or social groups.

[0072] In some embodiments, one or more of the embodiments, methods, approaches and / or technologies described above can be implemented in one or more computer programs or software applications that can be executed by a processor-based device or system. As an example, this processor-based system may include a processor-based device or system 900, or a computer, an entertainment system, a game console, a graphics workstation, a server, a client, a portable device, a device similar to a tablet computer, etc. This type of computer program can be used to perform various steps and / or features of the method and / or technology described above. In other words, the computer program can be suitable for making the device or system based on the processor perform and realize the function described above or configure the device or system to perform and realize the function described above. For example, this type of computer program can be used to implement any embodiment of the method, step, technology or feature described above. As another example, this type of computer program can be used to implement any type of tool or similar utility tool using any one or more of the embodiments, methods, approaches and / or technologies described above. In some embodiments, program code macros, modules, loops, subroutines, calls, etc. within or outside the computer program can be used to perform various steps and / or features of the method and / or technology described above. In some embodiments, the computer program may be stored or included on one or more computer-readable storage or recording media, such as any of the one or more computer-readable storage or recording media described herein.

[0073] Thus, in some embodiments, the present invention provides a computer program product comprising a medium for including a computer program for computer input; and a computer program included in the medium, the computer program for causing a computer to perform or execute steps, the steps comprising any one or more steps involved in any one or more of the embodiments, methods, approaches and / or techniques described herein. For example, in some embodiments, the present invention provides one or more non-transitory computer-readable storage media storing one or more computer programs, the one or more computer programs being suitable for causing a processor-based device or system to perform steps, the steps comprising: identifying one or more objects in one or more real-world scene images associated with a user; adding the identified one or more objects to a real-world object list associated with the user; assigning an object category to each object in the real-world object list based on image recognition; and providing a notification to the user that a content item has been associated with an object category assigned to one of the objects on the real-world object list associated with the user. In another example, in some embodiments, the present invention provides one or more non-transitory computer-readable storage media storing one or more computer programs suitable for causing a processor-based device or system to perform steps, the steps comprising: identifying one or more objects in one or more images associated with a first user, wherein the identification is performed by an image recognition technique executed by the processor-based device; for each identified object, searching through one or more images associated with other users to identify one or more images of interest, wherein the one or more images of interest each depict an object that (i) has an appearance that substantially matches the identified object and (ii) has been specified by one of the other users to be included in the search; generating a notification based at least in part on any identified image of interest; and providing the notification to the first user. In another example, in some embodiments, the present invention provides one or more non-transitory computer-readable storage media storing one or more computer programs suitable for causing a processor-based device or system to perform steps comprising: identifying one or more objects in one or more real-world scene images associated with a first user; adding the identified one or more objects to a real-world object list associated with the first user; assigning an object category to each object in the real-world object list based on image recognition; receiving content items for real-world objects in the real-world object list from the first user; and storing the content items and object categories of the real-world objects on a server so that the content items can be retrieved by a second user viewing an object in the object category of the real-world object.

[0074] Although the invention disclosed herein has been described with reference to specific embodiments and applications thereof, numerous modifications and variations may be made thereto by those skilled in the art without departing from the scope of the invention as set forth in the claims.

Claims

1. A method for identifying an object, the method comprising: assigning object categories to real-world objects displayed in an image associated with the first user based on the object recognition; as well as providing a notification to the first user that a content item provided by a second user has been associated with the object category; wherein the second user tags the content item to a real-world object displayed in an image associated with the second user, and wherein the content item is different from the image associated with the second user.

2. The method of claim 1, further comprising: The real-world object displayed in the image associated with the first user is identified based on object recognition.

3. The method of claim 1, further comprising: The real-world object displayed in the image associated with the first user is added to a list of real-world objects associated with the first user.

4. The method of claim 3, wherein: The real-world object list is configured to be edited by the first user.

5. The method according to any one of claims 1 to 4, wherein: The object category corresponds to one or more of: a unique object, a group of visually similar objects, and a group of objects that share one or more common attributes.

6. The method according to any one of claims 1 to 4, wherein: The providing the notification is based on one or more of a location of a user device, a sharing setting, and a social network connection between the first user and the second user.

7. A device for identifying an object, the device comprising: processor-based devices; as well as A non-transitory storage medium storing a set of computer-readable instructions configured to cause the processor-based device to perform a plurality of steps, the steps comprising: assigning object categories to real-world objects displayed in the image associated with the first user based on the object recognition; and providing a notification to the first user that a content item provided by a second user has been associated with the object category; wherein the second user tags the content item to a real-world object displayed in an image associated with the second user, and wherein the content item and the image associated with the second user are different.

8. The device of claim 7, wherein: The set of computer readable instructions is further configured to cause the processor-based device to perform a plurality of steps comprising: The real-world object displayed in the image associated with the first user is identified based on object recognition.

9. The device of claim 7, wherein: The set of computer readable instructions is further configured to cause the processor-based device to perform a plurality of steps comprising: The real-world object displayed in the image associated with the first user is added to a list of real-world objects associated with the first user.

10. A method for identifying an object, the method comprising: assigning object categories to real-world objects displayed in an image associated with the first user based on the object recognition; receiving, from the first user, a content item for tagging to the real-world object, the content item being different from the image associated with the first user; tagging the received content item to the real-world object; as well as The content item and the object category of the real-world object are stored on a server such that the content item is retrievable by a second user viewing an object in the object category of the real-world object.

11. The method of claim 10, further comprising: The real-world object displayed in the image associated with the first user is identified based on object recognition.

12. The method of claim 10, further comprising: The real-world object displayed in the image associated with the first user is added to a list of real-world objects associated with the first user.

13. The method according to any one of claims 10 to 12, further comprising: Settings are received from the first user, wherein the settings further restrict who is allowed to retrieve the content item.

14. The method of claim 13, wherein: The settings include whether the real-world object will be matched only with images of the real-world object or with images of any object that shares one or more common attributes with the real-world object; as well as Assigning the object class to the real-world object is based at least in part on the setting.

15. The method according to any one of claims 10 to 12, wherein: The object category corresponds to one or more of: a unique object, a group of visually similar objects, and a group of objects that share one or more common attributes.

16. A device for identifying an object, comprising: processor-based devices; as well as A non-transitory storage medium storing a set of computer-readable instructions configured to cause the processor-based device to perform a plurality of steps, the steps comprising: assigning object categories to real-world objects displayed in an image associated with the first user based on the object recognition; receiving, from the first user, a content item for tagging to the real-world object, the content item being different from the image associated with the first user; tagging the received content item to the real-world object; and The content item and the object category of the real-world object are stored on a server such that the content item is retrievable by a second user viewing an object in the object category of the real-world object.

17. The apparatus of claim 16, wherein: The set of computer readable instructions is further configured to cause the processor-based device to perform a plurality of steps comprising: The real-world object displayed in the image associated with the first user is identified based on object recognition.

18. The apparatus of claim 16, wherein: The set of computer readable instructions is further configured to cause the processor-based device to perform a plurality of steps comprising: The real-world object displayed in the image associated with the first user is added to a list of real-world objects associated with the first user.

19. The apparatus of any one of claims 16 to 18, wherein: The set of computer readable instructions is further configured to cause the processor-based device to perform a plurality of steps comprising: Settings are received from the first user, wherein the settings further restrict who is allowed to retrieve the content item.

20. The apparatus of claim 19, wherein: The settings include whether the real-world object will be matched only with images of the real-world object or with images of any object that shares one or more common attributes with the real-world object; as well as Assigning the object class to the real-world object is based at least in part on the setting.

Images