Method and system for automatically guided image capture and presentation

Abstract

A system including a computer-readable medium carrying at least one program and a system for facilitating automatically guided image capture and display are proposed. In some embodiments, the method includes: capturing a first image representing an item; classifying a portion of pixels in the first image as background pixels; modifying the first image based on the classification to generate a second image representing the item; displaying the second image on a display; receiving data defining graffiti on the displayed second image; classifying a second portion of pixels in the second image as background pixels based on the graffiti; modifying the second image based on the classification to generate a third image representing the item; and displaying the third image on the display.

Description

[0001] This application is a divisional application of invention patent application 201680070578.1 entitled “Automatically Guided Image Capture and Display”, which entered the Chinese national phase of PCT national application No. PCT / US2016 / 064057 with an application date of November 30, 2016.

[0002] priority

[0003] This patent application claims priority to U.S. Provisional Patent Application Serial No. 62 / 263,433, filed on December 4, 2015, and U.S. Patent Application Serial No. 15 / 361,973, filed on November 28, 2016, the entire contents of which are incorporated herein by reference. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Some embodiments are illustrated by way of example and not limitation in the accompanying drawings.

[0005] Figure 1 is a block diagram illustrating functional components of guided image capture in a presentation system, according to some example embodiments.

[0006] Figure 2 is a flow diagram illustrating the operation of a device in performing a method for automatically guided image capture and presentation, according to some example embodiments.

[0007] Figure 3 is a flow diagram illustrating the operation of a device in performing a method for automatically guided image capture and presentation, according to some example embodiments.

[0008] Figure 4 is a flow diagram illustrating the operation of a device in performing a method for automatically guided image capture and presentation, according to some example embodiments.

[0009] Figure 5 is a diagram illustrating a method for automatically guided image capture presentation, according to some example embodiments.

[0010] Figure 6 is a network diagram illustrating a network environment suitable for facilitating automatically guided image capture and presentation, according to some example embodiments.

[0011] Figure 7 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium and performing any one or more of the methodologies discussed herein, according to some example embodiments. DETAILED DESCRIPTION

[0012] Reference will now be made in detail to specific example embodiments for implementing the subject matter of the present invention. Examples of these specific embodiments are shown in the accompanying drawings. It should be understood that these examples are not intended to limit the scope of the claims to the illustrated embodiments. On the contrary, they are intended to cover substitutions, modifications and equivalents that may be included within the scope of the present disclosure. In the following description, specific details are set forth in order to provide a thorough understanding of the subject matter. Embodiments may be implemented without some or all of these specific details.

[0013] Example embodiments described herein facilitate automatically guided image capture and presentation of item lists (e.g., item data records). Mobile device logic can perform automatically guided image capture and presentation independent of network connectivity, allowing for enhanced creation of item lists and improved efficiency of network resources. Embodiments facilitate image guidance, automatic background removal, automatic completion of item list forms, and online store management.

[0014] In an example embodiment, a user of a mobile device (e.g., a seller) can use the mobile device to automatically create an item listing. The guided image capture and display system on the mobile device guides the user to capture an image of the item using the camera of the mobile device, detects and automatically removes the background portion of the image, retains only the image portion including the item of interest, and allows the user to further modify the image by manually changing the background portion of the image.

[0015] In another example embodiment, the guided image capture and display system facilitates automatic completion of an item list form by inferring information from the captured image (e.g., category recognition, feature recognition such as color, pattern, and style). Automatic completion of the item list form may also include inferring certain keywords associated with the item and title prediction from the captured item image. The system may also use multiple images in the automatic completion of the item list. Automatic completion of the item list may also include displaying to the user a list of items that may have similar titles and / or prices, thereby helping the user to construct the title of the item list and the price of the item. The guided image capture and display system may also allow the user to display items from the item list in an augmented reality environment, the augmented reality environment including a representation layer for displaying items on a virtual shelf or hanger for viewing by potential buyers visiting the user's online store. Based on the information inferred from the captured image, the guided image capture and display system may automatically select a representation layer according to the category and / or characteristics of the item (e.g., displaying a virtual shelf for shoes, displaying a virtual hanger for clothes). The user of the mobile device may also manually select a representation layer for the item. Additionally, the augmented reality environment may allow the user to customize the presentation layer (e.g., allowing the user to select a style of virtual shelves, hangers, or wallpaper based on the user's preferences or target audience).

[0016] Figure 11 is a block diagram illustrating functional components of a guided image capture and presentation system (GIPS) 124, according to some example embodiments. As described above, 124 facilitates automated guided image capture and presentation of item lists. Various functional modules of GIPS 124 are illustrated, wherein GIPS 124 is provided as part of a network system that communicates information about Figure 6 As will be understood by those skilled in the art of computer and Internet related technologies, Figure 1 Each component (e.g., module or engine) shown in the may be implemented using hardware (e.g., a processor of a machine) or a combination of logic (e.g., executable software instructions) and hardware for executing logic (e.g., a memory and processor of a machine).

[0017] GTPS 124 is shown to include image guidance module 104, background removal module 106, presentation module 108, list module 210, and database 112, all configured to communicate with each other (e.g., via a bus, shared memory, switch, or application programming interface (API)). In addition, various modules of GIPS 124 can access database 122 via a database server, and each of the various modules of GIPS 124 can communicate with one or more third-party applications (e.g., a social networking platform). Although Figure 1 The modules of the present invention are discussed in the singular sense, but it is understood that in other embodiments, multiple modules may be employed.

[0018] The image guidance module 104 assists the user of the mobile device in capturing images of items for a list of items. For example, the image guidance module 104 guides the user to position the mobile device in an optimal direction relative to the items, detect the items through the camera of the mobile device, and capture one or more image frames of the items. The image guidance module 104 estimates the background portion of the image and the foreground portion of the image for each captured image frame. For example, the estimation of the background and foreground portions of the image can be achieved by: identifying a first area and a second area; and determining that at least one area within the first area includes a group of pixels that is different (e.g., different in color) from a group of pixels in the second area, and vice versa. When identifying areas of different pixel groups, the image guidance module 104 estimates that the first area or the second area includes an item of interest. In addition to estimating the location of the item of interest (e.g., a location within the first area or the second area), the image guidance module 104 conveys instructions for capturing an image of the item to the user of the mobile device. In some embodiments, the user interface of the mobile device displays a viewfinder (e.g., a shape superimposed on the user interface of the mobile device) for guiding the positioning of the item for image capture.

[0019] When the user holds the mobile device at a certain distance from the object and moves the mobile device, the image guidance module 104 can display human-readable instructions (e.g., text instructions) within the user interface, instructing the user to move the mobile device so that the object will be completely included within the boundaries of the viewfinder. For example, the inner boundary of the viewfinder can correspond to the aforementioned first area (e.g., foreground) and the outer boundary can correspond to the aforementioned second area (e.g., background). In other embodiments, the image guidance module 104 does not instruct the user to move the mobile device, but instead displays instructions instructing the user to increase or decrease lighting or reposition the object, such as removing an object from the background portion of the image or separating two objects within the foreground portion of the image (e.g., an object of interest is close to another object). In some embodiments, the guided image capture processes each captured image frame and instructs the user of the mobile device in real time. The captured images can be stored on the mobile device, such as in a database (e.g., database 112) or a location within a memory, and can also be stored in a remote network-based system (e.g., a network-based system similar to that regarding Figure 6 described system) or in memory

[0020] The background removal module 106 automatically removes the background portion of the image, leaving only the image portion containing the item of interest. Similar to the guided image capture described above, background removal can be performed in real time on a mobile device. In addition, the background module 106 can also automatically remove the background portion of an image frame or a group of image frames. In some embodiments, the background removal module 106 accesses the image frame of the captured item and detects the boundary of the item in the image to separate the item from the image background. When detecting the boundary of the item, the background removal module 106 performs an initial estimate of a binary mask, which includes a first area and a second area of ​​the image, such as the first area and the second area described above with respect to the image guidance module 104. For both the first area and the second area, the background removal module 106 calculates a color histogram and uses the color histogram to calculate the probability of each pixel. When calculating the probability of each pixel, the background removal module 106 creates a probability map. The background removal module 106 can smooth the probability map using a domain transform geodesic filter and obtain a binary mask using a threshold selection method. Threshold selection methods assign one binary value to pixels having values ​​(eg, RGB values) above a determined threshold, and another binary value to pixels having values ​​below the determined threshold.

[0021] The background removal module 106 can also perform manual editing of the image by receiving user input on the mobile device. In some embodiments, a binary mask of the image is created, and the user of the mobile device manually edits the binary mask by creating a doodle to further define the outline of the item of interest and further separate the background portion of the image from the item of interest. A doodle is defined as a certain type of user input created by a user in the following manner: bringing an object into a predetermined proximity with the user interface of the mobile device (e.g., touching the screen of the mobile device with a finger), thereby causing a portion of the user interface (e.g., representing an image) to be highlighted or otherwise distinguished from the rest of the image. After receiving the user input doodle, the background removal module 106 uses domain transform geodesic filtering to smooth the doodle. The background removal module 106 scales the smoothed doodle to a value above or below a threshold. For example, the smoothed doodle is scaled to a value above or below a determined threshold corresponding to the threshold used in the threshold selection method described above for automatic background removal. The scaled doodle is added or subtracted from the probability map created during automatic background removal (depending on whether the value of the doodle is above or below the determined threshold), and a binary mask is obtained from the probability map. Background removal module 106 may also cause the graffiti to shrink in size or grow in size to cover the portion of the background to be removed. In addition, background removal module 106 may also allow the graffiti to capture the boundaries of the item of interest within the detected image. In some embodiments, the binary mask obtained after scaling of the graffiti is unique relative to the binary mask created during automatic background removal.

[0022] The list module 110 automatically fills out an item list form to create an item list. For example, the automatic item form list is completed by inferring information from an image of the item (e.g., including characteristics of the item such as color, style, category, brand name, etc.). In some embodiments, the information about the item is inferred from an image or image frame captured during the guided image capture described above with respect to the image guidance module 104. In terms of automatically completing the item list form, the list module 110 also includes displaying to the user a list of items that may have similar titles and / or prices, thereby assisting the user in constructing the title of the item list and the price of the item. When the item list form is completed, the item list is created, including the image of the item, the information about the item inferred from the image, and any additional input information received from the user of the mobile device. The guided image capture and display system 124 uses the completed item list to display the item within an augmented reality environment of the seller's online store.

[0023] The display module 108 enables items (e.g., captured item images as described above) to be displayed in an augmented reality environment including a representation layer. For example, the display module 108 can display items on virtual shelves or hangers for viewing by potential buyers who visit the user's online store. In some embodiments, based on the information of the item inferred from the image, the list module 110 automatically selects a representation layer (e.g., a virtual shoe rack for displaying shoes, a virtual hanger for displaying clothing, or a virtual shelf) according to the category and / or characteristics of the item. The user of the mobile device can also manually select a representation layer for the item. In addition, the augmented reality environment allows the user to customize the representation layer (e.g., allowing the user to select the style of the virtual shelf, hanger, or wallpaper based on the user's preferences or target audience). To enhance the augmented reality experience for viewers of the online store (e.g., visiting buyers or sellers), the display module 108 allows the addition of virtual sales indicators, such as "red sales tags". The virtual sales indicator allows the seller to attract the attention of viewers of the online store to a specific item, for example, indicating that the items in the seller's online store are being offered for sale at a specific price. In some embodiments, when viewing the presentation layer of a seller's online store, a viewer can select a displayed item (e.g., by touching an item on a virtual shelf or hanger displayed on a user interface of a mobile device) and upon selecting the item, presentation module 108 links the viewer to a page displaying detailed information about the item.

[0024] Figure 2 2 is a flow chart illustrating the operation of a device in performing a method 200 for automatically guided capture and presentation of images, according to some example embodiments. The method 200 includes capturing an image of an item 202, automatically removing an image background 204, performing manual mask editing 206, generating an item list 208 using inferred image information, and presenting the item list in an augmented reality environment 210. The method 200 may be embodied in computer readable instructions for execution by one or more processors, such that the steps of the method 200 may be performed in part or in whole by functional components (e.g., modules) of the GIPS 124: Therefore, the method 200 is described below by way of example with reference to the functional components of the GIPS 124. However, it should be understood that the method 200 may be deployed on various other hardware configurations and is not intended to be limited to the functional components of the GIPS 124.

[0025] In operation 202, GIPS 124 assists a user of a mobile device in capturing images of items for a list of items. For example, GIPS 124 guides the user to position the mobile device in an optimal direction relative to the items, detect the items through a camera of the mobile device, and capture one or more image frames of the items. For example, GIPS 124 estimates a background portion of the image and a foreground portion of the image for each captured image frame, such as by: identifying a first region and a second region; and determining that at least one region within the first region includes a set of pixels that is different (e.g., different in color) from a set of pixels in the second region, or vice versa. GIPS 124 estimates that the first region or the second region includes an item of interest. In addition, GIPS 124 communicates instructions to the user of the mobile device to capture an image of the item. When the user holds the mobile device, GIPS 124 displays human-readable instructions (e.g., text instructions) within a user interface, instructing the user to move the mobile device so that the item will be completely included within the boundaries of the viewfinder, instructing the user to increase or decrease lighting, or repositioning the object, such as removing an object from a background portion of the image or separating two objects in a foreground portion of the image (e.g., the item of interest is close to another object). When the GIPS 124 detects that the item is completely contained within the boundaries of the viewfinder, an image is captured.

[0026] At operation 204, GIPS 124 automatically removes the image background. When automatically removing the image background, GIPS 124 leaves only the portion of the image that includes the item of interest. As an example of the sub-operations involved in operation 204, Figure 33 is a flow chart illustrating the operation of a device in performing a method for automatically guided image capture and presentation according to some example embodiments. As shown, the operation of automatically removing the image background includes: guessing a binary mask 302, calculating a color histogram 304, calculating a probability map 306, smoothing the probability map 308, and obtaining a binary mask 310. In operation 302, GIPS 124 makes an initial guess of a binary mask. GIPS 124 accesses a captured image of an object and detects the boundaries of the object within the image, separating the object from the image background. When detecting the boundaries of the object, GIPS 124 makes an initial estimate of a binary mask, which includes a first region and a second region of the image. In operation 304, GIPS 124 calculates a color histogram for the first region and the second region of the image. In operation 306, GIPS 124 calculates a probability map. GIPS 124 uses the color histogram to calculate the probability of each pixel. When calculating the probability of each pixel, GIPS 124 creates a probability map. In operation 308, the GIPS 124 smoothes the probability map by using domain transform geodesic filtering. When performing domain transform geodesic filtering, the GIPS 124 assigns one binary value to pixels having values ​​(e.g., RGB values) above a determined threshold and another binary value to pixels having values ​​below the determined threshold. When assigning values ​​to pixels above or below the determined threshold, the GIPS 124 obtains a binary mask in operation 310.

[0027] Back to Figure 2 At operation 206, GIPS 124 performs manual mask editing. As an example of sub-operations involved in operation 206, Figure 4 is a flow diagram illustrating the operation of a device in performing a method for automatically guided image capture and presentation, according to some example embodiments.

[0028] In operation 402, GIPS 124 accesses a probability map (e.g., a probability map created during automatic image background removal). In operation 404, GIPS 124 receives user input in the form of a scribble on a mobile device. Using the scribble, GIPS 124 further defines the outline of the item of interest and further separates the background portion of the image from the item of interest. In operation 406, after receiving the user input scribble, GIPS 124 uses domain transform geodesic filtering to smooth the scribble. In operation 408, GIPS 124 scales the smoothed scribble to a value above or below a threshold. For example, the smoothed scribble may be scaled to a value above or below a determined threshold corresponding to a threshold used in creating a binary mask using a threshold selection method. In operation 410, GIPS 124 distinguishes the scaled scribble from the probability map by adding or subtracting (depending on whether the scribble's value is above or below a determined threshold) the scaled scribble value from the probability map created during automatic background removal. When distinguishing the scaled scribble from the probability map, in operation 412, GIPS 124 obtains a binary mask from the probability map. GIPS 124 can shrink or increase the size of the graffiti to cover the part of the background to be removed. In addition, GIPS 124 can also make the graffiti capture the boundaries of the items of interest in the detected image.

[0029] Back to Figure 2 In operation 208, GIPS 124 generates an item list using the inferred image information. GIPS 124 automatically populates an item list form to create the item list, inferring information (e.g., characteristics of the item including color, style, category, brand name) from the image of the item. GIPS 124 infers information about the item from the image or image frame captured during the guided image capture described with respect to operation 202. In automatically completing the item list form, GIPS 124 also includes displaying to the user a list of items that may have similar titles and / or prices, thereby assisting the user in constructing the title of the item list and the price of the item. GIPS 124 completes the item list form, creating the item list, including: the image of the item, the information about the item inferred from the image, and any additional input information received from the user of the mobile device. GIPS 124 uses the completed item list to display the item within an augmented reality environment of the seller's online store.

[0030] In operation 210, GIPS 124 displays the list of items in an augmented reality environment. GIPS 124 displays items from the list of items (e.g., captured item images as described above) in an augmented reality environment that includes a representation layer, for example, displaying the items on a virtual shelf or hanger for viewing by potential buyers visiting the user's online store. In some embodiments, based on information inferred from the image of the item, GIPS 124 automatically selects a representation layer (e.g., a virtual shoe rack displaying shoes, a virtual hanger displaying clothes, or a virtual shelf) according to the category and / or characteristics of the item. In order to increase the augmented reality experience of viewers of the online store (e.g., visiting buyers or sellers), GIPS 124 allows the addition of virtual sales indicators, allowing sellers to attract the attention of viewers from the online store to specific items. GIPS 124 allows viewers to select a displayed item (e.g., by touching an item on a virtual shelf or hanger displayed on a user interface of a mobile device), and upon selecting the item, GIPS 124 links the viewer to a page displaying detailed information about the item.

[0031] Figure 5 is a diagram illustrating a method of automatically guiding an image capture demonstration according to some example embodiments. In 502, GIPS 124 assists a user of a mobile device in capturing images of items of a list of items. Figure 5 As shown, GIPS 124 guides the user to position the mobile device in an optimal direction relative to the item. For example, GIPS 124 estimates the background portion of the image and the foreground portion of the image for each captured image frame by: identifying a first region and a second region; and determining that at least one region within the first region includes a set of pixels that is different (e.g., a different color) from a set of pixels in the second region, and vice versa. GIPS 124 estimates that the first region or the second region includes the item of interest. Figure 5 As shown, for example, GIPS 124 indicates to the user the estimated first and second regions of the image by displaying the outline of the item of interest. In addition, GIPS 124 communicates instructions to the user of the mobile device to capture an image of the item. When the user holds the mobile device, GIPS 124 displays instructions on the user interface to instruct the user to move the mobile device so that the item will be completely included within the boundaries of the viewfinder, to instruct the user to increase or decrease lighting or reposition the object. When GIPS 124 detects that the item is completely included within the boundaries of the viewfinder, the image is captured.

[0032] At 504, GIPS 124 automatically removes the image background. Figure 5As shown, GIPS 124 leaves only the portion of the image that includes the item of interest and displays a viewfinder showing the boundaries of the item of interest, which may have been used during guided image capture. GIPS 124 makes an initial guess at a binary mask and detects the boundaries of the item within the image, separating the item from the image background. GIPS 124 computes a color histogram probability map, using the color histogram to calculate the probability of each pixel. GIPS 124 smoothes the probability map by using domain transform geodesic filtering. While generating the image in 504, GIPS 124 may also perform manual mask editing, receiving user input in the form of scribbles and further defining the outline of the item of interest, and further separating the background portion of the image from the item of interest.

[0033] At 506, GIPS 124 generates an item list using the inferred image information. GIPS 124 automatically populates an item list form to create an item list, inferring information (e.g., characteristics of the item including color, style, category, brand name) from the image of the item. GIPS 124 infers information about the item from the image or image frame captured during the guided image capture described with respect to operation 202. Figure 5 As shown in 510 of the embodiment, GIPS 124 also includes displaying suggested titles or categories or lists of items that may have similar titles and / or prices to the user, thereby assisting the user in constructing the titles of the item list and the prices of the items. GIPS 124 completes the item list form and creates the item list, including the image of the item, information about the item inferred from the image, and any additional information received from the user of the mobile device. GIPS 124 uses the completed item list to display the item within the augmented reality environment of the seller's online store. At 508, GIPS 124 displays the item list in the augmented reality environment. Figure 5 As shown, GIPS 124 displays items in an augmented reality environment that includes a presentation layer, for example, displaying items on a virtual shelf for viewing by potential buyers visiting the user's online store.

[0034] Figure 6 6 is a network diagram showing a network system 600 suitable for facilitating automated guided image capture and presentation having a client-server architecture configured for exchanging data over a network 602, according to one embodiment. Although the network system 600 is depicted as having a client-server architecture, the inventive subject matter is certainly not limited to such an architecture and may equally find application in, for example, event-driven, distributed, or peer-to-peer architecture systems. Furthermore, in order to avoid obscuring the subject matter of the present invention in unnecessary detail, Figure 6Various functional components that are not germane to conveying an understanding of the subject matter of the present invention are omitted. Furthermore, it should be understood that although various functional components of network system 600 are discussed in the singular sense, multiple instances of any of the various functional components may be used.

[0035] The network system 600 includes a network-based content publisher 604 in communication with a client device 606 and a third-party server 608. In some example embodiments, the network-based content publisher 604 may be a network-based marketplace (e.g., eBay.com). The network-based content publisher 604 communicates and exchanges data related to various functions and aspects associated with the network system 600 and its users within the network system 600. The network-based content publisher 604 may provide server-side functionality to network devices (such as client devices 606) via a network 602 (e.g., the Internet).

[0036] Client devices 606 may be operated by users using network system 600 to exchange data over network 602. Such data exchanges include transmitting (delivering) data to users of network system 600, receiving data from users of network system 600, and processing data related to content and users of network system 600. Data may include, but is not limited to, images; video or audio content; user preferences; product and service feedback, suggestions and reviews; product, service, manufacturer and supplier recommendations and identifiers; product and service listings associated with buyers and sellers; product and service advertisements; auction bids; transaction data; user profile data; and social data, among others.

[0037] The client device 606 can interface with the web-based content publisher 104 via a connection to the network 602. Depending on the form of the client device 606, any of various types of connections and networks 602 can be used. For example, the connection can be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other types of cellular connections. Such a connection can implement any of a variety of types of data transmission technologies, such as single carrier wireless transmission technology (1xRTT), evolution data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate evolution for GSM (EDGE) technology, or other data transmission technology (e.g., fourth generation wireless, 4G network). When such technology is adopted, the network 602 can include a cellular network of cellular sites with overlapping geographical coverage interconnected by multiple cellular telephone switches. These cellular telephone switches can be coupled to a network backbone (e.g., a public switched telephone network (PSTN), a packet switched data network, or other types of networks).

[0038] In another example, the connection to network 602 may be a wireless fidelity (Wi-Fi, IEEE 802.11x type) connection, a Worldwide Interoperability for Microwave Access (WiMAX) connection, or other type of wireless data connection. In such an embodiment, network 602 may include one or more wireless access points coupled to a local area network (LAN), a wide area network (WAN), the Internet, or another packet-switched data network. In yet another example, the connection to network 602 may be a wired connection (e.g., an Ethernet link), and network 602 may be a LAN, a WAN, the Internet, or another packet-switched data network. Therefore, a variety of different configurations are clearly contemplated.

[0039] In various embodiments, the data exchanged within the network system 600 may depend on user-selected functionality available through one or more client or user interfaces (UIs). The UI may be associated with a client device, such as a client device 606 executing a network client 610 (e.g., an Internet browser), which may communicate with a network-based content publisher 604. The UI may also be associated with one or more applications 612 executing on the client device 606, such as a mobile application designed to interact with the network-based content publisher 604 or a social networking platform hosted by a third-party server 608.

[0040] In particular, the web-based content publisher 604, API server 614, and web server 616 are coupled to the application server 618 and provide programming and web interfaces to the application server 618, respectively. Figure 6 As shown, application server 618 can be coupled to network 602 via API server 614 and network server 616, for example, via a wired or wireless interface. Application server 618 is further shown coupled to database server 620 that facilitates access to database 622. In some examples, application server 618 can directly access database 622 without requiring database server 620. Database 622 can include multiple databases that can be internal or external to network-based content publisher 604.

[0041] For example, the application server 618 may host one or more applications that provide a number of content publishing and viewing functions and services to users accessing the network-based content publisher 604. For example, the network-based content publisher 604 may host a marketplace application that provides a number of marketplace functions and services to users, such as publishing, listing, and price setting mechanisms whereby sellers may list (or publish) (information related to) goods or services (also collectively referred to as "products") for sale, buyers may express interest in such goods or services or indicate a willingness to purchase such goods or services, and prices may be set for transactions related to the goods or services. The client device 606 (e.g., a mobile device) hosts the GIPS 124 that performs the automatically guided capture and presentation of images.

[0042] Messages generated and published by GIPS 124 may also include references to content (e.g., links). As used herein, the terms "content" or "content item" refer to electronic data consumed by a viewer (e.g., a user) on a display, client device, or page / display-based media (e.g., World Wide Web (WWW) media embodied in a browser and accessible via the Internet). As such, the terms "content" and "content item" may refer to data associated with readable text, data associated with images, data associated with graphics or video, programmatic content, scripts, or data corresponding to various combinations of these. Furthermore, while Figure 6 6. In the embodiment shown in FIG. 6 , GIPS 124 is shown forming part of client device 606, but it will be appreciated that in alternative embodiments, GIPS 124 may form part of a separate and distinct service from client device 606. For example, in some embodiments, GIPS 124 may be hosted in whole or in part by application server 618.

[0043] Database 622 stores data about various functions and aspects associated with network system 600 and its users. For example, database 622 stores and maintains user profiles of users of network-based content publisher 604. Each user profile includes user profile data describing aspects of a particular user. For example, user profile data may include demographic data, user preferences, social data, and financial information. For example, demographic data may include information describing one or more characteristics of a user, such as gender, age, location information (e.g., hometown or current location), employment history, education history, contact information, family relationships, or user interests. For example, financial information may include private financial information of a user, such as account number, credentials, password, device identifier, user name, phone number, credit card information, bank information, transaction history records, or other financial information that facilitates online transactions for a user.

[0044] The database 622 also stores records of user activities. Thus, the network-based content publisher 604 can monitor, track, and record the activities of users who interact with various components of the user network system 600 using one or more devices (e.g., client devices 606) during a user session. Each user session can be maintained in a repository stored in the database 622. Thus, user activity data can include past keyword searches that users have performed, web pages viewed by each user, products added to a user's wish list or watch list, products added to an electronic shopping cart, and products owned by a user. According to some embodiments, the records of user activities are linked to user profile data to maintain an association of the user profile with the activities that the corresponding user has performed.

[0045] In the case where the network-based content publisher 104 is a network-based marketplace, the database 122 stores product information. Such product information may include, for example, a product identifier (e.g., title or model number), price, brand, manufacturer, model number, brand name, text description, size, style, product dimensions, compatibility information, or any other information that may be used to describe a product. In these cases, the database 622 may also store transaction histories of users of the network-based content publisher 604, which transaction history includes information related to transactions that may be offered for sale by merchants utilizing the marketplace services provided by the network-based content publisher 604. The transaction history information may include, for example, a description of a product offered for sale, sold, or purchased by a user, an identifier of the product, a category to which the product belongs, a purchase price, a purchase date, a purchase quantity, a bid quantity for the product, or various combinations thereof.

[0046] Figure 6 Also shown is a third-party application 626 executing on the third-party server 608 that can provide information or services to users of the application server 618 or the client device 606. The third-party application 626 programmatically accesses the network-based content publisher 604 via a programming interface provided by the API server 614. The third-party application 626 is associated with any organization that can transact with or provide services to the application server 618 or the client device 606. For example, the third-party application 626 can be associated with a network-based social networking platform (e.g., etc.) are associated.

[0047] Figure 7is a block diagram illustrating components of a machine 700 that is capable of reading instructions 724 from a machine-readable medium 722 (e.g., a non-transitory machine-readable medium, a machine-readable storage medium, a computer-readable storage medium, or any suitable combination thereof) and performing, in whole or in part, any one or more of the methodologies discussed herein, according to some example embodiments. Specifically, Figure 7 A machine 700 is shown in the example form of a computer system (e.g., a computer) in which instructions 724 (e.g., software, programs, applications, applet, applications, or other executable code) may be executed to cause the machine 700 to perform, in whole or in part, any one or more of the methodologies discussed herein.

[0048] In alternative embodiments, the machine 700 operates as a standalone device or can be communicatively coupled (e.g., networked) to other machines. In a networked deployment, the machine 700 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a distributed (e.g., peer-to-peer) network environment. The machine 700 can be a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a cellular phone, a smart phone, a set-top box (STB), a personal digital assistant (PDA), a network appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 724 specifying actions to be performed by the machine. In addition, although only a single machine 700 is shown, the term "machine" should also be understood to include any collection of machines that execute instructions 724 individually or in combination to perform any one or more of the methods discussed herein in whole or in part.

[0049] The machine 700 includes a processor 702 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any suitable combination thereof), a main memory 704, and a static memory 706, which are configured to communicate with each other via a bus 708. The processor 702 contains solid-state digital microcircuits (e.g., electronic, optical, or both) that are temporarily or permanently configurable by some or all of the instructions 724, so that the processor 702 can be configured to perform any one or more of the methods described herein in whole or in part. For example, a collection of one or more microcircuits of the processor 702 can be configured to execute one or more modules (e.g., software modules) described herein. In some example embodiments, the processor 702 is a multi-core CPU (e.g., a dual-core CPU, a quad-core CPU, or a 128-core CPU), where each of the multiple cores behaves as an independent processor capable of performing any one or more of the methods discussed herein in whole or in part. Although the benefits described herein may be provided by a machine 700 having at least a processor 702, these same benefits may be provided by a different kind of machine that does not include a processor (e.g., a purely mechanical system, a purely hydraulic system, or a hybrid mechanical-hydraulic system) if such a processor-free machine is configured to perform any one or more of the methods described herein.

[0050] The machine 700 may also include a graphics display 710 (e.g., a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, a cathode ray tube (CRT), or any other display capable of displaying graphics or video). The machine 700 may also include an alphanumeric input device 712 (e.g., a keyboard or keypad), a cursor input device 714 (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, an eye tracking device, or other pointing device), a data storage 716, an audio generation device 718 (e.g., a sound card, an amplifier, a speaker, a headphone jack, or any suitable combination thereof), and a network interface device 720.

[0051] The data store 716 (e.g., a data storage device) includes a machine-readable medium 722 (e.g., a tangible and non-transitory machine-readable storage medium) on which are stored instructions 724 that embody any one or more of the methods or functions described herein. The instructions 724 may also reside, in whole or in part, within the main memory 704, within the processor 702 (e.g., within a cache memory of the processor), or both, before or during execution by the machine 700. Thus, the main memory 704 and the processor 702 may be considered machine-readable media (e.g., a tangible and non-transitory machine-readable medium). The instructions 724 may be transmitted or received over the network 190 via the network interface device 720. For example, the network interface device 720 may communicate the instructions 724 using any one or more transmission protocols (e.g., the Hypertext Transfer Protocol (HTTP)).

[0052] In some example embodiments, the machine 700 may be a portable computing device (e.g., a smartphone, tablet, or wearable device) and have one or more additional input components 730 (e.g., sensors or meters). Examples of such input components 730 include image input components (e.g., one or more cameras), audio input components (e.g., one or more microphones), direction input components (e.g., compasses), position input components (e.g., global positioning system (GPS) receivers), orientation components (e.g., gyroscopes), motion detection components (e.g., one or more accelerometers), height detection components (e.g., altimeters), biometric input components (e.g., heart rate detectors or blood pressure detectors), and gas detection components (e.g., gas sensors). Input data collected by any one or more of these input components 730 may be accessible and available for use by any of the modules described herein.

[0053] As used herein, the term "memory" refers to a machine-readable medium capable of temporarily or permanently storing data, and may be considered to include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, and cache memory. Although the machine-readable medium 722 is shown as a single medium in the example embodiment, the term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database or associated cache and server) capable of storing instructions 724. The term "machine-readable medium" should also be understood to include any medium or combination of multiple media capable of storing instructions 724 for execution by the machine 700, so that the instructions 724, when executed by one or more processors (e.g., processor 702) of the machine 700, cause the machine 700 to perform any one or more of the methods described herein in whole or in part. Therefore, "machine-readable medium" refers to a single storage device or device, as well as a cloud-based storage system or storage network including multiple storage devices or devices. Accordingly, the term "machine-readable medium" should be understood to include, but is not limited to, one or more tangible and non-transitory data repositories (e.g., data volumes), optical disks, magnetic disks, or any appropriate combination thereof in the example form of solid-state memory chips. As used herein, "non-transitory" machine-readable media specifically does not include propagating signals per se. In some example embodiments, instructions 724 executed by machine 700 may be conveyed by a carrier medium. Examples of such carrier media include storage media (e.g., non-transitory machine-readable storage media such as solid-state memory that physically moves from one place to another) and transitory media (e.g., propagating signals conveying instructions 724).

[0054] Certain example embodiments are described herein as including modules. A module may constitute a software module (e.g., code stored or otherwise embodied in a machine-readable medium or in a transmission medium), a hardware module, or any suitable combination thereof. A "hardware module" is a tangible (e.g., non-transitory) physical component (e.g., a collection of one or more processors) that is capable of performing certain operations and that may be configured or arranged in some physical manner. In various example embodiments, one or more computer systems or one or more of their hardware modules may be configured by software (e.g., an application or portion thereof) as a hardware module that operates to perform the operations described herein for the module.

[0055] In some example embodiments, the hardware module may be implemented mechanically, electronically, hydraulically, or any suitable combination thereof. For example, the hardware module may include dedicated circuits or logic that are permanently configured to perform certain operations. The hardware module may be or include a dedicated processor, such as a field programmable gate array (FPGA) or an ASIC. The hardware module may also include programmable logic or circuits that are temporarily configured by software to perform certain operations. As an example, the hardware module may include software contained within a CPU or other programmable processor. It should be understood that the decision to implement the hardware module mechanically, hydraulically, in a dedicated and permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations.

[0056] Thus, the phrase "hardware module" should be understood to include a tangible entity that can be physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. In addition, as used herein, the phrase "hardware-implemented module" refers to a hardware module. Considering an example embodiment in which a hardware module is temporarily configured (e.g., programmed), each hardware module need not be configured or instantiated at any one time. For example, in the case where a hardware module includes a CPU that is configured by software to become a special-purpose processor, the CPU can be configured as correspondingly different special-purpose processors (e.g., each included in a different hardware module) at different times. Software (e.g., software modules) can configure one or more processors accordingly, such as becoming or otherwise constituting a specific hardware module at one time and becoming or otherwise constituting different hardware modules at different times.

[0057] A hardware module can provide information to other hardware modules and receive information from other hardware modules. Therefore, the described hardware modules can be considered to be communicatively coupled. In the case of multiple hardware modules at the same time, communication can be achieved by signal transmission (e.g., through circuits and buses) between two or more hardware modules. In an embodiment in which multiple hardware modules are configured or instantiated at different times, communication between these hardware modules can be achieved, for example, by storing and retrieving information in a memory structure accessible to multiple hardware modules. For example, a hardware module can perform an operation and store the output of the operation in a memory (e.g., a memory device) to which it is communicatively coupled. Afterwards, another hardware module can access the memory later to retrieve and process the stored output. The hardware module can also initiate communication with an input or output device, and can operate on a resource (e.g., a collection of information from a computing resource).

[0058] The various operations of the example methods described herein may be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions described herein. As used herein, a "processor-implemented module" refers to a hardware module in which the hardware includes one or more processors. Therefore, the operations described herein may be at least partially processor-implemented, hardware-implemented, or both, since a processor is an example of hardware, and at least some of the operations within any one or more of the methods discussed herein may be performed by one or more processor-implemented modules, hardware-implemented modules, or any appropriate combination thereof.

[0059] In addition, such one or more processors may perform operations in a "cloud computing" environment or as a service (e.g., in a "software as a service" (SaaS) implementation). For example, at least some of the operations within any one or more of the methods discussed herein may be performed by a group of computers (e.g., as an example of a machine including a processor), which may be accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs)). The performance of certain operations may be distributed among one or more processors, whether resident only within a single machine or deployed across multiple machines. In some example embodiments, one or more processors or hardware modules (e.g., modules implemented by a processor) may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, one or more processors or hardware modules may be distributed across multiple geographic locations.

[0060] Throughout the specification, multiple instances can be implemented as components, operations or structures described as single instances. Although the individual operations of one or more methods are illustrated and described as individual operations, one or more of the individual operations can be performed simultaneously, and it is not required to perform the operations in the order shown. The structures and their functions shown as individual components and functions in the example configuration can be implemented as combined structures or components with combined functions. Similarly, the structures and functions shown as individual components can be implemented as individual components and functions. These and other changes, modifications, additions and improvements fall within the scope of this paper theme.

[0061] Some parts of the subject matter discussed herein may be presented in terms of an algorithm or symbolic representation of the operation of data stored in a memory (e.g., a computer memory or other machine memory) as a bit or binary digital signal. Such an algorithm or symbolic representation is an example of a technique used by a person of ordinary skill in the field of data processing to convey the essence of their work to other persons of ordinary skill in the art. As used herein, an "algorithm" is a self-consistent sequence of operations or similar processes that result in a desired result. In this case, the algorithm and operation involve the physical manipulation of physical quantities. Typically, but not necessarily, such an amount may be in the form of an electrical, magnetic or optical signal that can be stored, accessed, transmitted, combined, compared or otherwise manipulated by a machine. Mainly for commonly used reasons, it is convenient to refer to such signals using words such as "data", "content", "bit", "value", "element", "symbol", "character", "term", "numerical value", "digital". However, these words are just convenient labels and are associated with appropriate physical quantities.

[0062] Unless otherwise noted, discussions herein using words such as "access," "process," "detect," "compute," "determine," "generate," "exhibit," "display," and the like refer to actions or processes that can be performed by a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or any suitable combination thereof), registers, or other machine components that receive, store, transmit, or display information. In addition, unless otherwise specifically stated, the terms "a" or "an" are used herein, as is common in patent documents, to include one or more than one instance. Finally, as used herein, the conjunction "or" refers to a non-exclusive "or" unless otherwise specifically stated.

[0063] Various examples of the methods, machine-readable media, and systems (eg, machines, devices, or other apparatus) discussed herein are described below.

[0064] Example

[0065] 1. A method, comprising capturing an image of an object, comprising: guiding a user to position a mobile device in a direction relative to the object so that the object is included within the boundaries of an area of ​​a user interface of the mobile device; detecting the object through a camera of the mobile device and capturing at least one image frame of the object; estimating boundaries of a first area and boundaries of a second area of ​​the at least one image frame; determining that the first area includes a larger portion of the object than the second area; displaying instructions on the user interface of the mobile device, instructing the user to take an action so that the object will become completely included within and visible within the boundaries of the first area; and detecting that the object is completely included within and visible within the boundaries of the first area.

[0066] 2. The method according to Example 1 also includes automatically removing the background of the image frame, including: accessing the image frame of the object; detecting the boundary of the object within the first area, the boundary of the object is along the outline of the object on the user interface, the inner part of the boundary includes only the object, and the outer part of the boundary includes the background of the image frame; distinguishing the inner part of the boundary from the outer part of the boundary; estimating a first binary mask, the first binary mask includes the inner part of the boundary of the image frame and the outer part of the boundary; calculating a color histogram of the image frame including the inner part of the boundary and the outer part of the boundary; calculating a probability map of the image frame including the inner part of the boundary and the outer part of the boundary; and smoothing the probability map including the inner part of the boundary and the outer part of the boundary.

[0067] 3. The method according to Example 2 also includes obtaining a second binary mask, assigning a first binary value to each pixel of a first plurality of pixels in the image frame having an RGB value higher than a determined threshold, and assigning a second binary value to each pixel of a second plurality of pixels in the image frame having an RGB value lower than the determined threshold, and obtaining a first background-free object image, and applying the second binary mask to the image frame.

[0068] 4. The method of any one of Examples 1 to 3, further comprising performing manual mask editing, comprising accessing the probability map, receiving user input from the mobile device, the user input comprising scribbles, smoothing the scribbles and scaling the scribbles, comprising assigning the first binary value to each pixel of a third plurality of pixels including the scribbles and having an RGB value above the determined threshold, and assigning the second binary value to each pixel having an RGB value below the determined threshold.

[0069] 5. The method according to any one of Examples 1 to 4 further includes obtaining a third binary mask, comprising: distinguishing the scaled graffiti from the probability map, adding or subtracting the binary values ​​of the pixels including the scaled graffiti from the probability map, and obtaining a second background-free object image, applying the third binary mask to the first background-free object image.

[0070] 6. The method according to any one of Examples 1 to 5 further includes generating an item list, inferring item information from the image frame and automatically applying the inferred item information to an item list form.

[0071] 7. The method according to any one of Examples 1 to 6 further includes displaying a list of items in an augmented reality environment, including displaying the items from the image frame in a presentation layer on a user interface of a mobile device.

[0072] 8. A system comprising one or more processors and executable instructions accessible on a computer-readable medium, which when executed cause the one or more processors to perform operations, the operations comprising: capturing an image of an object, comprising: guiding a user to position a mobile device in a direction relative to the object so that the object is included within the boundaries of an area of ​​a user interface of the mobile device; detecting the object through a camera of the mobile device and capturing at least one image frame of the object; estimating boundaries of a first area and boundaries of a second area of ​​the at least one image frame; determining that the first area includes a larger portion of the object than the second area; displaying instructions on the user interface of the mobile device instructing the user to take action so that the object will become completely included within the boundaries of the first area and visible within the boundaries of the first area; and detecting that the object is completely included within the boundaries of the first area and visible within the boundaries of the first area.

[0073] 9. A system according to Example 8, wherein the operation also includes: automatically removing the background of the image frame, which includes: accessing the image frame of the object; detecting the boundary of the object within the first area, the boundary of the object is along the outline of the object on the user interface, the inner part of the boundary includes only the object, and the outer part of the boundary includes the background of the image frame; distinguishing the inner part of the boundary from the outer part of the boundary; estimating a first binary mask, the first binary mask includes the inner part of the boundary of the image frame and the outer part of the boundary; calculating a color histogram of the image frame including the inner part of the boundary and the outer part of the boundary; calculating a probability map of the image frame including the inner part of the boundary and the outer part of the boundary; and smoothing the probability map including the inner part of the boundary and the outer part of the boundary.

[0074] 10. A system according to Example 9, wherein the operation also includes obtaining a second binary mask, assigning a first binary value to each pixel in a first plurality of pixels in the image frame having an RGB value above a determined threshold, and assigning a second binary value to each pixel in a second plurality of pixels in the image frame having an RGB value below the determined threshold, and obtaining a first background-free object image, applying the second binary mask to the image frame.

[0075] 11. A system according to any one of Examples 8 to 10, wherein the operations also include performing manual mask editing, including accessing the probability map, receiving user input from the mobile device, the user input including scribbles, smoothing the scribbles and scaling the scribbles, including assigning the first binary value to each pixel of a third plurality of pixels including the scribbles and having an RGB value above the determined threshold, and assigning the second binary value to each pixel having an RGB value below the determined threshold.

[0076] 12. A system according to Example 11, wherein the operation also includes obtaining a third binary mask, including distinguishing the scaled graffiti from the probability map, adding or subtracting binary values ​​of pixels including the scaled graffiti from the probability map, and obtaining a second background-free object image, applying the third binary mask to the first background-free object image.

[0077] 13. A system according to any one of Examples 8 to 12, wherein the operation also includes generating an item list, inferring item information from the image frame and automatically applying the inferred item information to an item list form.

[0078] 14. A system according to any one of Examples 8 to 13, wherein the operation also includes presenting a list of items in an augmented reality environment, including displaying the items from the image frame in a presentation layer on a user interface of the mobile device.

[0079] 15. A computer-readable hardware storage device having a set of program instructions stored therein, which, when executed by a computer, cause the computer to perform operations, the operations comprising: capturing an image of an object, comprising: guiding a user to position a mobile device in a direction relative to the object so that the object is included within the boundaries of an area of ​​a user interface of the mobile device; detecting the object through a camera of the mobile device and capturing at least one image frame of the object, estimating a boundary of a first area and a boundary of a second area of ​​the at least one image frame; determining that the first area includes a larger portion of the object than the second area; displaying instructions on the user interface of the mobile device, instructing the user to take action so that the object will become completely included within the boundaries of the first area and visible within the boundaries of the first area; and detecting that the object is completely included within the boundaries of the first area and visible within the boundaries of the first area.

[0080] 16. The computer-readable hardware storage device according to Example 15 further includes automatically removing the background of the image frame, including: accessing the image frame of the object; detecting the boundary of the object within the first area, the boundary of the object is along the outline of the object on the user interface, the inner part of the boundary includes only the object, and the outer part of the boundary includes the background of the image frame; distinguishing the inner part of the boundary from the outer part of the boundary; estimating a first binary mask, the first binary mask includes the inner part of the boundary of the image frame and the outer part of the boundary; calculating a color histogram of the image frame including the inner part of the boundary and the outer part of the boundary; calculating a probability map of the image frame including the inner part of the boundary and the outer part of the boundary; and smoothing the probability map including the inner part of the boundary and the outer part of the boundary.

[0081] 17. The computer-readable hardware storage device of Example 16 further includes obtaining a second binary mask, assigning a first binary value to each pixel of a first plurality of pixels in the image frame having an RGB value higher than a determined threshold, and assigning a second binary value to each pixel of a second plurality of pixels in the image frame having an RGB value lower than the determined threshold, and obtaining a first background-free object image, applying the second binary mask to the image frame.

[0082] 18. A computer-readable hardware storage device according to any one of Examples 15 to 17, further comprising performing manual mask editing, including accessing the probability map, receiving user input from the mobile device, the user input comprising scribbles, smoothing the scribbles and scaling the scribbles, including assigning the first binary value to each pixel of a third plurality of pixels including the scribbles and having an RGB value above the determined threshold, and assigning the second binary value to each pixel having an RGB value below the determined threshold.

[0083] 19. The computer-readable hardware storage device of Example 18 further includes obtaining a third binary mask, including distinguishing the scaled graffiti from the probability map, adding or subtracting binary values ​​of pixels including the scaled graffiti from the probability map, and obtaining a second background-free object image, applying the third binary mask to the first background-free object image.

[0084] 20. A computer-readable hardware storage device according to any one of Examples 15 to 19, further comprising generating an item list, inferring item information from the image frame and automatically applying the inferred item information to an item list form.

[0085] 21. A carrier medium carrying a set of program instructions, which, when executed by a computer, causes the computer to perform the method described in any one of Examples 1 to 7.

Claims

1. A method for creating an item list, comprising: capturing a first image representing an item; Classifying some pixels in the first image as background pixels; modifying the first image based on the classification to generate a second image representing the item; displaying a second image on a display; receiving data defining graffiti on a displayed second image; smoothing the scribble using domain transform geodesic filtering, and scaling the smoothed scribble to a value above or below a threshold; adding or subtracting the scaled scribble from a probability map created during generation of the second image to distinguish the scaled scribble from the probability map, obtaining a binary mask from the probability map, thereby classifying a second portion of pixels in the second image as background pixels; modifying the second image based on the classification to generate a third image representing the item; as well as The third image is displayed on the display.

2. The method according to claim 1, wherein: Classifying a portion of pixels in the first image as background pixels includes: for each pixel in the portion of pixels, determining a probability of whether the corresponding pixel is part of the background of the first image.

3. The method according to claim 2, further comprising: A color histogram of the first image is determined, wherein the determination of the probability is based on the color histogram.

4. The method according to claim 1, further comprising: Guidelines are displayed to position the mobile device in an orientation relative to the item such that the item is included within the boundaries of an area of ​​a user interface of the mobile device.

5. The method according to claim 1, further comprising: identifying a category of an object represented by a third image; and generating an item list of the items according to the identified categories.

6. The method according to claim 1, wherein: Displaying the second image includes: displaying a binary mask of the second image, and Receiving data defining the scribble includes receiving input defining one or more changes to the binary mask.

7. The method according to claim 1, wherein: Classifying some pixels in the first image as background pixels includes generating a binary mask of the first image and smoothing the binary mask, wherein the modification of the first image is based on the smoothed binary mask.

8. A system for creating an item list, comprising: one or more processors; as well as executable instructions accessible on a computer-readable medium that, when executed, cause the one or more processors to perform operations comprising: capturing a first image representing an item; Classifying some pixels in the first image as background pixels; modifying the first image based on the classification to generate a second image representing the item; displaying a second image on a display; receiving data defining graffiti on a displayed second image; smoothing the scribble using domain transform geodesic filtering, and scaling the smoothed scribble to a value above or below a threshold; obtaining a binary mask from the probability map by adding or subtracting the scaled scribble from the probability map created during the generation of the second image, thereby classifying a second portion of pixels in the second image as background pixels; modifying the second image based on the classification to generate a third image representing the item; and The third image is displayed on the display.

9. The system according to claim 8, wherein: Classifying a portion of pixels in the first image as background pixels includes: for each pixel in the portion of pixels, determining a probability of whether the corresponding pixel is part of the background of the first image.

10. The system according to claim 9, wherein: The operations also include determining a color histogram of the first image, wherein the determination of the probability is based on the color histogram.

11. The system according to claim 8, wherein: The operations also include displaying guidance to position the mobile device in an orientation relative to the item such that the item is included within the boundaries of an area of ​​a user interface of the mobile device.

12. The system according to claim 8, wherein: The operations also include identifying a category of the item represented by the third image; and generating an item list for the item according to the identified category.

13. The system according to claim 8, wherein: Displaying the second image includes: displaying a binary mask of the second image, and Receiving data defining the scribble includes receiving input defining one or more changes to the binary mask.

14. The system according to claim 8, wherein: Classifying some pixels in the first image as background pixels includes generating a binary mask of the first image and smoothing the binary mask, wherein the modification of the first image is based on the smoothed binary mask. 15 . A computer-readable storage medium having a set of program instructions stored thereon, wherein when the program instructions are executed by a computer, the computer is caused to execute the method according to claim 1 .

Images