Augmented reality self-scanning and self-checkout

Abstract

A method of providing a shopping user interface for a head-mounted display device includes detecting a product barcode in a field of view of the head-mounted display device. Product information and pricing information for the item are retrieved based on information in the barcode. The product information and the pricing information may include an item identifier and a price displayed in a first display area of the head-mounted display device. After the default number of transient delays can be adjusted, the item identifier and the price are transmitted into a shopping cart list display area of the head-mounted display device.

Description

[0001] Related application data

[0002] This application claims priority to U.S. patent application No. 17 / 830,661, filed on June 2, 2022. Technical Field

[0003] The present disclosure relates generally to display devices and, more particularly, to display devices for augmented reality. Background Art

[0004] The head mounted device may be implemented as a transparent or translucent display through which a user of the head mounted device can view the surrounding environment. Such a device enables the user to look through the transparent or translucent display to view the surrounding environment, and also to see objects (e.g., virtual objects such as 3D renderings, images, videos, text, etc.) generated for display that appear to be part of the surrounding environment and / or superimposed on the surrounding environment. This is generally referred to as "augmented reality" or "AR".

[0005] Users of head mounted devices can access and use computer software applications to perform various tasks or participate in entertainment activities. In order to use the computer software application, the user interacts with a user interface provided by the head mounted device. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0007] Figure 1 is a perspective view of a head mounted device according to some examples.

[0008] Figure 2 According to some examples Figure 1 Another view of the headset.

[0009] Figure 3 It is shown that according to some examples, Figure 1 A block diagram of a networking system 300 showing details of a head mounted device.

[0010] Figure 4 A view through a lens of a head-mounted device including a shopping user interface is shown according to some examples.

[0011] Figure 5 A view through a lens of a head-mounted device including a shopping user interface is shown according to some examples.

[0012] Figure 6 A view through a lens of a head-mounted device including a shopping user interface is shown according to some examples.

[0013] Figure 7 A view through a lens of a head-mounted device including a shopping user interface is shown according to some examples.

[0014] Figure 8 A view through a lens of a head-mounted device including a shopping user interface is shown according to some examples.

[0015] Fig. 9 A view through a lens of a head-mounted device including a shopping user interface is shown according to some examples.

[0016] Fig.10 A view through a lens of a head-mounted device including a shopping user interface is shown according to some examples.

[0017] Fig.11 A view through a lens of a head-mounted device including a shopping user interface is shown according to some examples.

[0018] Fig.12 A view through a lens of a head-mounted device including a shopping user interface is shown according to some examples.

[0019] Fig.13 is a flow chart illustrating a method of providing a shopping user interface according to some examples.

[0020] Fig.14 is shown according to some examples Fig.14 A flowchart of a subroutine of the flowchart.

[0021] Fig.15 is a diagrammatic representation of a networking environment in which the present disclosure may be deployed, according to some examples.

[0022] Fig.16 is a block diagram illustrating a software architecture within which the present disclosure may be implemented, according to some examples.

[0023] Fig.17 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0024] Some head-mounted AR devices, such as AR glasses, include a transparent or translucent display, which enables the user to see through the transparent or translucent display to view the surrounding environment. Additional information or objects (e.g., virtual objects such as 3D renderings, images, videos, text, etc.) are displayed on the display and appear as part of the surrounding environment and / or superimposed on the surrounding environment to provide the user with an augmented reality (AR) experience. The display may, for example, include a waveguide that receives a beam from a projector, but any suitable display for presenting augmented content or virtual content to the wearer may be used.

[0025] As referred to herein, the phrase "augmented reality experience" includes or refers to various image processing operations corresponding to image modification, filtering, media overlay, transformation, etc., as further described herein. In some examples, these image processing operations provide an interactive experience of a real-world environment, in which objects, surfaces, backgrounds, lighting, etc. in the real world are enhanced by computer-generated perceptual information. In this context, an "augmented reality effect" includes a collection of data, parameters, and other assets for applying a selected augmented reality experience to an image or video feed. In some examples, augmented reality effects are provided by Snap Inc. under the registered trademark LENSES.

[0026] During grocery shopping or other retail shopping, the user is typically required to gather all items together and perform a checkout transaction at a self-serve checkout counter or at a conventional checkout point near the entrance of the store. There have been some attempts to implement checkout-free shopping, such as by scanning RFID tags on products as customers leave the store, but these have not been widely adopted. Smartphone applications can also provide barcode scanning, pricing and other information, but this requires the user to hold the smartphone in one hand while holding, for example, a shopping basket in the other hand, which requires one of the items to be put down or moved away when handling the items to be purchased.

[0027] The head-mounted augmented reality device can provide a retail experience in which relevant product information, pricing and checkout lists can be conveniently created and managed. The camera in the head-mounted augmented reality device can identify products to be purchased, obtain and display pricing information, and maintain a list of items that have been selected for purchase.

[0028] In some examples, a computer-implemented method using a head-mounted display device including a camera and one or more displays includes: detecting the presence of an optical code in a field of view of a camera of the head-mounted display device; obtaining product and pricing information for an item based on information in the optical code, the product and pricing information including an item identifier and a price; displaying the item identifier and price in a first display area of ​​one or more displays of the head-mounted display device; and transmitting the item identifier and price to a shopping cart list in a second display area of ​​the one or more displays of the head-mounted display device.

[0029] The item identifier and price may be transmitted to the shopping cart list after a certain delay and without user input. The head mounted display device may include a touch pad located on a side of the head mounted display device, the method further comprising: displaying a default item quantity for the item in the first display area; and receiving an input on the touch pad (such as a forward or backward swipe) to change the default item quantity for the item to an updated item quantity.

[0030] The product information of the item can be compared with the product information in the user profile. Based on the comparison, if it is determined that the item is not preferred, a warning can be displayed in the first display area. The method can also include setting a default item quantity for the item to zero based on determining that the item is not preferred.

[0031] The computer-implemented method may also include: in the case where the head-mounted display device includes a touch pad located on a side of the head-mounted display device, the method also includes: receiving a user selection of an item identifier in the shopping cart list; displaying the item identifier and price in the first display area; and receiving user input on the touch pad (such as swiping forward or backward) to change the item quantity of the item to an updated item quantity.

[0032] The computer-implemented method may further include: determining a store identification; and accessing product availability and pricing information based on the store identification. The store identification may be determined from the group consisting of scanning a store identification optical code located at the store, a short-range RF signal transmitted at the store, and a GPS location of the store.

[0033] In some examples, a non-transitory computer-readable storage medium is provided, which includes instructions that, when executed by a computer, cause the computer to perform operations according to any of the above methods, the operations being used to provide a user interface in a head-mounted display device including a camera and one or more displays, the methods including but not limited to the following method, in which the operations include: detecting the presence of an optical code in a field of view of a camera of the head-mounted display device; obtaining product and pricing information for an item based on information in the optical code, the product and pricing information including an item identifier and a price; displaying the item identifier and price in a first display area of ​​one or more displays of the head-mounted display device; and transmitting the item identifier and price to a shopping cart list in a second display area of ​​one or more displays of the head-mounted display device.

[0034] In some examples, a computing device is provided that includes a processor and a memory storing instructions that, when executed by the processor, configure the device to perform operations for providing a user interface in a head-mounted display device that includes a camera and one or more displays according to any of the above methods, the methods including, but not limited to, the following method, in which the operations include: detecting the presence of an optical code in a field of view of a camera of the head-mounted display device; obtaining product and pricing information of an item based on information in the optical code, the product and pricing information including an item identifier and a price; displaying the item identifier and price in a first display area of ​​one or more displays of the head-mounted display device; and transmitting the item identifier and price to a shopping cart list in a second display area of ​​one or more displays of the head-mounted display device.

[0035] Other technical features will be readily apparent to those skilled in the art from the accompanying drawings, descriptions and claims.

[0036] Figure 11 is a perspective view of a head-mounted AR display device (e.g., glasses 100) according to some examples. Glasses 100 may include a frame 102, which is made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first optical element holder or a left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or a right optical element holder 106 connected by a bridge 112. A first optical element or a left optical element 108 and a second optical element or a right optical element 110 may be provided in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be provided in the glasses 100.

[0037] The frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, the frame 102 can be formed from a single piece of material to have a unitary or integrated construction.

[0038] The glasses 100 may include a computing device such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 302 discussed below.

[0039] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0040] The glasses 100 include a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0041] In some examples, left camera 114 and right camera 116 provide video frame data for glasses 100 to use to extract 3D information from a real-world scene.

[0042] The glasses 100 may also include a touchpad 126 mounted to or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is generally arranged vertically, with the touchpad 126 being approximately parallel to the temple of the user in some examples. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, although potentially more vertical than the vertical. Additional user input may be provided by one or more buttons 128, which in the example shown are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. The one or more touchpads 126 and buttons 128 provide a means by which the glasses 100 can receive input from a user of the glasses 100.

[0043] Figure 2 The eyeglasses 100 are shown from the user's perspective. For clarity, the Figure 1 As shown in the figure, Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured within left and right optical element holders 104, 106, respectively.

[0044] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .

[0045] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the right eye of the user to provide an image on or in the right optical element 110 that is superimposed on the view of the real world seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the left eye of the user to provide an image on or in the left optical element 108 that is superimposed on the view of the real world seen by the user. The combination of the GPU, the front optical assembly 202, the left optical element 108, and the right optical element 110 provides an optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay of the user's real-world view, including displaying a 3D user interface to the user of the glasses 100.

[0046] However, it should be understood that other display technologies or configurations may be used within the optical engine to display images to the user in the user's field of view. For example, instead of the projector 204 and waveguide, an LCD, LED or other display panel or surface may be provided.

[0047] In use, the user of the glasses 100 will be presented with information, content, and various 3D user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, associated devices (e.g., Figure 3 ) and / or hand movements, position, and orientation detected by the glasses 100 to interact with the glasses 100.

[0048] Figure 3 is a block diagram showing a networked system 300 including details of the glasses 100 according to some examples. The networked system 300 includes the glasses 100, a user device 328, and a server system 332. The user device 328 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low power wireless connection 336 and / or a high speed wireless connection 334. The user device 328 is connected to the server system 332 via a network 330. The network 330 may include any combination of wired and wireless connections. The server system 332 may be one or more computing devices that are part of a service or network computing system. The server system 332 may be a server system 332 that is a networked computing system that may be used separately. Fig.16 and Fig.17 The details of the software architecture 1604 or machine 1700 described in the implementation of the user device 328 and any elements of the server system 332 and network 330.

[0049] The glasses 100 include a data processor 302, a display 310, one or more cameras 308, and additional input / output elements 316. The input / output elements 316 may include microphones, audio speakers, biometric sensors, additional sensors, or additional display elements integrated with the data processor 302. Fig.16 and Fig.17 Examples of input / output elements 316 are further discussed. For example, input / output elements 316 may include any I / O component 1706, including output component 1728, motion component 1736, etc. Figure 2 Examples of display 310 are discussed in . In the particular examples described herein, display 310 includes displays for the left and right eyes of a user.

[0050] Data processor 302 includes image processor 306 (eg, video processor), GPU and display driver 338, tracking module 340, interface 312, low power circuitry 304, and high speed circuitry 320. Components of data processor 302 are interconnected by bus 342.

[0051] Interface 312 refers to any source of user commands provided to data processor 302. In one or more examples, interface 312 is a physical button that, when pressed, sends a user input signal from interface 312 to low-power processor 314. Pressing such a button and then releasing it immediately can be processed by low-power processor 314 as a request to capture a single image, and vice versa. Pressing such a button for a first period of time can be processed by low-power processor 314 as a request to capture video data when the button is pressed and end video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, interface 312 can be any mechanical switch or physical interface capable of accepting user input associated with requesting data from camera 308. In other examples, interface 312 can have a software component, or can be associated with a command received wirelessly from another source, such as from user device 328.

[0052] Image processor 306 includes circuitry for receiving signals from camera 308 and processing those signals from camera 308 into a format suitable for storage in memory 324 or for transmission to user device 328. In one or more examples, image processor 306 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 308, and volatile memory used by the microprocessor in operation.

[0053] The low power circuit system 304 includes a low power processor 314 and a low power wireless circuit system 318. These elements of the low power circuit system 304 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 314 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 314 can accept user input signals from the interface 312. The low power processor 314 can also be configured to receive input signals or command communications from the user device 328 via the low power wireless connection 336. The low power wireless circuit system 318 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 318. In other examples, other low power communication systems may be used.

[0054] High-speed circuit system 320 includes a high-speed processor 322, a memory 324, and a high-speed wireless circuit system 326. High-speed processor 322 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 302. High-speed processor 322 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 334 using high-speed wireless circuit system 326. In some examples, high-speed processor 322 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Fig.16 The high-speed processor 322, which executes the software architecture of the data processor 302, manages data transmission with the high-speed wireless circuit system 326, in addition to any other duties. In some examples, the high-speed wireless circuit system 326 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 326 can implement other high-speed communication standards.

[0055] Memory 324 includes any storage device capable of storing camera data generated by camera 308 and image processor 306. Although memory 324 is shown as being integrated with high-speed circuitry 320, in other examples, memory 324 may be a separate, independent element of data processor 302. In some such examples, electrical wiring may provide a connection from image processor 306 or low-power processor 314 to memory 324 through a chip including high-speed processor 322. In other examples, high-speed processor 322 may manage addressing of memory 324 so that low-power processor 314 will direct high-speed processor 322 whenever a read or write operation involving memory 324 is desired.

[0056] The tracking module 340 estimates the position and orientation ("pose") of the glasses 100. For example, the tracking module 340 uses image data and corresponding inertial data from the camera 308 and the positioning component 1740, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world environment). The tracking module 340 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100 that indicates changes in relative position and orientation relative to physical objects in the real-world environment. The tracking module 340 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 310.

[0057] The GPU and display driver 338 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 310 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 338 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real environment.

[0058] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the user device 328 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 1606, such as the messaging application 1646.

[0059] Figure 4 A view 400 through a lens of a head mounted device, such as glasses 100, is shown according to some examples. View 400 represents what a user of glasses 100 sees and includes objects in the real world visible through optical elements 108 and optical elements 110 of glasses 100. In this case, the objects include products 404 available for purchase in a retail environment 402.

[0060] Superimposed on the retail environment 402 using at least one of the right near-eye display 206 and the left near-eye display 216 is a shopping user interface 406. The user interface 406 includes a marked (framed or shaded) scanning area 408 and a shopping cart 410, the scanning area being used to indicate where a product optical code (e.g., a barcode) can be scanned in the field of view of the glasses 100. The shopping cart 410 includes a list area 412 and a total cost area 414, which lists the total cost of the items in the shopping cart 410. The list area 412 can switch between a shopping list and a list of items that have been scanned by the user. The total cost area 414 includes a total cost 416 of the items in the shopping cart 410, and a checkout button 418 that can be selected when the user is ready to check out.

[0061] Upon arriving at the store, the user scans a QR or other machine-readable code or tag, such as an NFC tag, at the entrance of the store using the glasses 100 or the user device 328. The QR code includes information identifying the store. The QR code can be scanned, for example, by the user positioning the glasses 100 so that the QR code is within the scanning area 408. In other examples, other location or proximity alternatives such as Bluetooth beacons or GPS can be used to identify the store.

[0062] The user device 328 or glasses 100 uses the store's identifying information in the QR code or tag to obtain access to a list of products available at the store and the associated price of each product. The downloaded information includes an item identifier for each product, such as the item name and optionally a graphical representation of the item, such as a product photo, a generic item icon such as a milk carton, or a logo. The list can be downloaded to the user device 328 or glasses 100 for direct access, or can be made available for access as needed from a remote server such as database server 1516 or third party server 1508 (see Fig.15 ). Item listings can include relevant information such as loyalty program discounts, nutritional information, product graphics, special promotional considerations (e.g., buy two, get the third half off), and more.

[0063] Figure 5 A view through the lens of a head mounted device such as glasses 100 is shown according to some examples. View 500 shows that list area 412 is currently empty, but it includes a prompt for the user to view the barcode to scan the product. The user has moved a piece of cheese 502 into the field of view of glasses 100 so that its barcode 504 is within scanning area 408.

[0064] The barcode 504 is captured by one of the cameras 308, and its presence is detected and processed by the data processor 302 to identify and decode the barcode 504 information. The data processor 302 then compares the barcode information to the item list to obtain the relevant product information. In order to reduce the processing requirements on the glasses 100 and therefore reduce the power requirements of the glasses, the cameras are operated in a lower resolution computer vision mode that provides sufficient resolution to allow the barcode 504 to be recognized, but the resolution is lower than the resolution used for other image capture functions such as user-initiated photo or video capture. In addition, to further reduce power consumption, images for barcode detection are captured and scanned at a relatively low frequency, such as once every 1 to 3 seconds, which will provide sufficient responsiveness for the user experience.

[0065] After decoding the barcode 504, the glasses 100 or user device 328 uses the information from the barcode 504 to identify the cheese 502 from the list of products available at the store, including its price and any relevant information such as ingredients or loyalty program discounts or other discounts or promotions. The user interface 406 then transitions to Figure 6 Status shown.

[0066] Figure 6 A view through the lens of a head mounted device such as glasses 100 is shown according to some examples. View 600 shows that list area 412 is still currently empty of shopping, but information about cheese 502 (including its name 604, price 606, and product graphic 610) is shown in information box 602 in scan area 408.

[0067] Also shown is a quantity 608 and left and right arrows surrounding the quantity 608, indicating that the user can adjust the quantity 608 by swiping forward or backward on the touch pad 126 of the glasses 100. In response to a forward swipe, the quantity increases, while in response to a backward swipe, the quantity decreases. The selected quantity can then be confirmed by clicking on the touch pad 126. In the event that the user does not want to purchase the item, the quantity can be set to zero, and after the confirmation click, the information box 602 will disappear without the item appearing in the list area 412.

[0068] Alternatively, if the user wishes to purchase more than one item, they may temporarily remove the bar code 504 from the scanning area 408 and then return it to the scanning area 408 which would either increase the quantity 608 by one if the information box 602 still exists, or display a new information box 602 if the original information box 602 disappeared after the original item was transferred to the list area 412.

[0069] If the user wants to continue purchasing, after clicking on the confirmation button, Figure 7 As shown, information box 602 will disappear from scanning area 408 and the item will appear in list area 412 with any adjustments to the quantity. The item will also be removed from the associated shopping list. As previously described, the user can switch between shopping cart 410 and the shopping list to identify items that remain to be purchased. Shopping cart 410 is substantially the same as shopping cart 410, except that it is titled "Shopping List" and does not include a total cost area 414. The removal of items from the shopping list based on scanning and transferring items to shopping cart 410 can be based on matching specific items listed (e.g., specific brands, package sizes, and product versions) or can be based on general categories such as dishwashing detergent or cheese or eggs.

[0070] Figure 7 700 shows a view through a lens of a head mounted device such as glasses 100 according to some examples. Figure 6 Updated so that list area 412 includes item 702, which includes Figure 6 The scanned cheese 502 (including its name 604 and price 606) has been added to the cart 410, and the total cost area 414 has been updated to indicate the new total cost of the items in the cart 410.

[0071] Figure 8 800 shows a view through the lens of a head mounted device such as glasses 100 according to some examples. View 800 shows the list area 412 after several additional products have been scanned and added to the list area 412. Figure 7 User interface 406. The name, price, and graphic of each product are included, and the total cost area 414 has been updated to indicate the new total cost of the items in the shopping cart 410.

[0072] In this case, the scan area 408 includes an information box 802 indicating that an energy bar 806 has been scanned for potential purchase. However, in this case, the user's shopping profile specifies that the user is allergic to nuts, and by comparing the nutritional information of the energy bar 806 to the user's shopping profile, the glasses 100 or user device 328 has identified that the energy bar 806 may be an item that the user does not actually wish to purchase.

[0073] A warning 804 has been provided in the information box 802 indicating that the product contains peanuts, and a blank or zero quantity for the item has been indicated as the default value for the quantity 608. If the default zero quantity for the item is not overwritten due to user input on the touch pad 126 increasing the quantity 608 from zero, the information box 802 will disappear after a confirmation click on the touch pad 126, and the energy bar 806 will not appear in the list area 412.

[0074] Upon receiving user input to increase the quantity and confirming by clicking the touchpad 126, the information box 802 will disappear from the scanning area 408 and the energy bar 806 will appear in the list area 412 along with the other items.

[0075] Fig. 9 A view 900 through a lens of a head mounted device such as glasses 100 is shown according to some examples. View 900 shows a shopping cart 410 including several items but no scanned items and no product information box in the scanning area 408. The user may, for example, be ready to check out and review the items in the shopping cart 410 before checking out to verify completeness, adjust quantities, or remove items.

[0076] When an upward touch or downward touch is received by the user on the touch pad 126, the items displayed in the shopping cart 410 and the checkout button 418 may be highlighted for selection. The selection of the highlighted item (or the checkout button 418) is then confirmed by receiving a click input from the user on the touch pad 126. The selection of the highlighted item returns it to the scanning area 408 in the information box 602, as shown in FIG. Fig.10 shown.

[0077] Fig.10 A view 1000 through a lens of a head mounted device such as glasses 100 is shown according to some examples. The view 1000 is generated by receiving an up / down user input on the touch pad 126 to highlight an item, followed by receiving a confirmation click on the touch pad 126 to select the highlighted item, thereby selecting Fig. 9 Ricola tea items in shopping cart 410.

[0078] In response, an information box 1002 corresponding to Ricola tea is displayed in the scanning area 408. In this case, the user has decided not to purchase the item and adjusts the quantity 1004 to zero. In response to receiving a confirmation click on the touch pad 126, the information box 1002 is removed from the scanning area 408 and the corresponding item is removed from the shopping cart 410, as shown. Fig.11 shown.

[0079] Fig.11 A view 1100 through a lens of a head mounted device such as glasses 100 is shown according to some examples. The view 1100 is due to the Fig. 9 The view 1100 will be changed to the shopping cart 410 in question by removing the Ricola tea item. Fig.12 View shown.

[0080] Fig.12A view 1200 through a lens of a head mounted device such as glasses 100 is shown according to some examples. The view 1200 is selected by a user such as Fig. 9 or Fig.11 406 . View 1200 includes a checkout notification 1202 in user interface 406 , which provides a total amount due and a prompt to follow the instructions on the payment terminal at the front of the store. The list of items in shopping cart 410 can now be transmitted to the payment terminal, for example, via a short-range data transmission protocol such as Bluetooth or NFC or indirectly via server system 332 or third-party server 1508. In some examples, the user views a checkout terminal that includes a QR code containing information identifying the terminal. The terminal's identification information and the final list of items are then sent to server system 332 or third-party server 1508, which then sends the final list of items and the total amount, or just a payment prompt for the total amount, to the correct payment terminal.

[0081] The user then completes payment for the item using a known or future payment method, such as via a credit or debit card at the terminal or payment credentials stored in the glasses 100 or user device 328 .

[0082] Fig.13 1300 is a flowchart illustrating a method of providing a shopping user interface according to some examples. For illustrative purposes, the operations of flowchart 1300 are described herein as occurring serially or linearly. However, multiple operations of flowchart 1300 may occur in parallel. In addition, the operations of flowchart 1300 need not be performed in the order shown and / or one or more blocks of flowchart 1300 need not be performed and / or may be replaced by other operations.

[0083] Fig.13 The operations shown in will typically be performed on the glasses 100. In other examples, the operations are performed in conjunction between an application running on the user device 328 and related hardware and data processor 302 in or associated with the glasses 100. Various implementations are certainly possible in which some of the operations occur in the server system 332, or in which one application calls another application or SDK to obtain the desired functionality.

[0084] The method begins at operation 1302, where the glasses 100 run an application such as an applet or shopping application 1634 that provides shopping cart functionality (see Fig.16 ), which has been selected and launched by the user of the shopping application 1634 or mini program.

[0085] In operation 1302, a store logo is detected, for example, by the glasses scanning a QR code or NFT tag at the store entrance. In other examples, the glasses determine or receive other location or identification information by the glasses 100, for example, by determining a GPS location or receiving a transmission from an RF beacon. In operation 1304, the identification information in the QR code (or other identification or location information) is sent to a server such as a database server 1516 or a third-party server 1508 (see Fig.15 ) that provides access to a list of items available at the store and the price of each item. The list can be downloaded to the user device 328 or glasses 100 for direct access, or can be made available for access from the remote server as needed. The item list can include relevant information such as loyalty program discounts, nutritional information, product graphics, special promotional considerations (e.g., buy two, get the third half price), etc.

[0086] In operation 1306, the barcode 504 of the item is detected in the scanning area 408 in the field of view of the glasses 100. The barcode is decoded to determine product identification information such as a universal product code (UPC). Item information corresponding to the UPC is retrieved from the item list, and the item information is displayed in the user interface 406 by the glasses 100 in operation 1308. In some examples, the information is displayed in the information box 602.

[0087] In operations 1310, 1312, the item information is also compared to preferences, allergies, or other user-specific information in the user's profile, and any item information (e.g., ingredient warnings) that is specific to the user is included in the information displayed in operation 1308. In addition, any adjustments such as setting the quantity of items containing ingredients that cause allergies to zero are made in operation 1312. Special information displayed in operation 1312 may include promotions or discounts for items.

[0088] In operation 1314, any user input adjustment to the default quantity (typically one) is received by the glasses 100 and confirmed by clicking the touchpad 126. In operation 1316, any items with non-zero quantities and their associated quantities are added to the shopping cart 410. The total cost 416 in the total cost area 414 is updated to include the new addition. If no item adjustment is received in operation 1314, then after receiving a confirmation click on the touchpad or a certain amount of time (a short timer (e.g., three to five seconds) set by the glasses 100 expires, which is sufficient to allow the user to adjust the item quantity without disrupting the shopping experience) or scanning an additional barcode, any items with non-zero quantities and their associated quantities are added to the shopping cart 410 in operation 1320. The flowchart 1300 then continues at operation 1318, where the display of the item information from operation 1308 is removed from the user interface 406.

[0089] In operation 1322, additional user input is received. The additional input may include detecting an additional barcode in the scan area 408, selecting an item in the shopping cart 410 for quantity editing or removal, or selecting the checkout button 418. If an additional barcode is detected in operation 1322, the method returns to operation 1308 to display item information, and the method continues from there. If an item is selected, the flowchart continues to operation 1324. Fig.14 Operation 1402 in .

[0090] As described above, upon receiving a selection of the checkout button 418, in operation 1326, the glasses display a checkout prompt and a prompt to follow the instructions on the payment terminal at the front of the store, the checkout prompt providing the total amount due. In operation 1328, the list of items in the shopping cart is then transmitted to the payment terminal. The list of items in the shopping cart 410 can be sent to the payment terminal, for example, via a short-range data transmission protocol such as Bluetooth or NFC, or indirectly via a remote server. In some examples, the user views a checkout terminal that includes a QR code containing information identifying the terminal scanned by the glasses 100. The identification information of the terminal and the final list of items are then sent by the glasses 100 or the user device 328 to the remote server, which then sends the final list of items and the total amount, or a payment prompt for the total amount only, to the correct payment terminal.

[0091] In operation 1330 , the user then completes payment for the item using a known or future payment method, such as via a credit or debit card at a terminal or using payment credentials saved in the glasses 100 or the user device 328 .

[0092] Fig.141400 is a flowchart illustrating a method of providing a shopping user interface according to some examples. For illustrative purposes, the operations of flowchart 1400 are described herein as occurring serially or linearly. However, multiple operations of flowchart 1400 may occur in parallel. In addition, the operations of flowchart 1400 need not be performed in the order shown, and / or one or more blocks of flowchart 1400 need not be performed and / or may be replaced by other operations.

[0093] Fig.14 The operations shown in the flowchart 1400 will typically be performed on the glasses 100. In other examples, the operations are performed jointly between an application running on the user device 328 and related hardware and data processor 302 in or associated with the glasses 100. For the purpose of clarity, the flowchart 1400 is discussed herein with reference to such an example. Various implementations are of course possible in which some of the operations occur in the server system 332, or in which one application calls another application or SDK to obtain the desired functionality.

[0094] The method starts at operation 1402, and in the above operation, the display in operation 1308 is the same as in Fig.13 Item information corresponding to the item selected from the shopping cart 410 in operation 1322 is displayed. As previously described, the item information is also compared with the preferences, allergies, or other user-specific information in the user's profile in operation 1404, and the information displayed in operation 1406 includes any item information specific to the user. In addition, as previously described, the information displayed in operation 1406 includes any information about special offers or promotions or discounts associated with the selected item.

[0095] If any item quantity adjustments are received in operation 1408, then flow diagram 1400 continues to operation 1410. If no item adjustments are received in operation 1408, then display of the item information from operation 1308 is removed from user interface 406 after receiving a confirmation click on the touchpad.

[0096] In operation 1410, if the item quantity has been adjusted to zero and a confirmation click on the touch pad is received, the item is removed from the shopping cart in operation 1412. In operation 1416, the display of the item information is removed from the scanning area 408 and the flowchart returns to Fig.13 If the adjusted item quantity is not zero in operation 1410, then after receiving the confirmation click on the touchpad, the total cost 416 in the total cost area 414 is updated in operation 1414 to reflect the new quantity. The flowchart 1400 then continues at operation 1416, where the display of the item information is removed from the user interface 406, and the flowchart returns to Fig.13Operation 1322 in .

[0097] Fig.15 is a block diagram illustrating an example messaging system 1500 for exchanging data (e.g., messages and associated content) over a network. The messaging system 1500 includes multiple instances of user devices 328 that host several applications including a messaging client 1502 and other applications 1504. The messaging client 1502 is communicatively coupled to other instances of the messaging client 1502 (e.g., hosted on respective other user devices 328), a messaging server system 1506, and a third-party server 1508 via a network 330 (e.g., the Internet). The messaging client 1502 can also communicate with the locally hosted application 1504 using an application program interface (API).

[0098] The messaging clients 1502 are able to communicate and exchange data with other messaging clients 1502 and with a messaging server system 1506 via the network 330. The data exchanged between the messaging clients 1502 and between the messaging clients 1502 and the messaging server system 1506 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0099] The messaging server system 1506 provides server-side functionality to a particular messaging client 1502 via the network 330. Although some functionality of the messaging system 1500 is described herein as being performed by the messaging client 1502 or by the messaging server system 1506, it may be a design choice whether some functionality is located within the messaging client 1502 or within the messaging server system 1506. For example, it may be technically preferable to initially deploy some technologies and functionality within the messaging server system 1506, but then migrate the technologies and functionality to the messaging client 1502 where the user device 328 has sufficient processing power.

[0100] The messaging server system 1506 supports various services and operations provided to the messaging clients 1502. Such operations include transmitting data to the messaging clients 1502, receiving data from the messaging clients 1502, and processing data generated by the messaging clients 1502. As examples, the data may include message content, user device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 1500 is activated and controlled by functions available via the user interface (UI) of the messaging client 1502.

[0101] Turning now specifically to the messaging server system 1506, an application program interface (API) server 1510 is coupled to and provides a programming interface to an application server 1514. The application server 1514 is communicatively coupled to a database server 1516, which facilitates access to a database 1520 that stores data associated with messages processed by the application server 1514. Similarly, a web server 1524 is coupled to and provides a web-based interface to the application server 1514. To this end, the web server 1524 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0102] The application program interface (API) server 1510 receives and transmits message data (e.g., commands and message payloads) between the user device 328 and the application server 1514. Specifically, the application program interface (API) server 1510 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 1502 to activate the functions of the application server 1514. The application program interface (API) server 1510 exposes various functions supported by the application server 1514, including: account registration; login functionality; sending messages from a particular messaging client 1502 to another messaging client 1502 via the application server 1514, sending media files (e.g., images or videos) from the messaging client 1502 to the messaging server 1512, and possible access by another messaging client 1502; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the user device 328; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 1502).

[0103] The application server 1514 hosts several server applications and subsystems, including, for example, a messaging server 1512, an image processing server 1518, and a social network server 1522. The messaging server 1512 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 1502. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 1502. In view of the hardware requirements for such processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 1512.

[0104] The application server 1514 also includes an image processing server 1518 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 1512.

[0105] The social network server 1522 supports various social networking functions and services and makes these functions and services available to the messaging server 1512. To this end, the social network server 1522 maintains and accesses an entity graph within the database 1520. Examples of functions and services supported by the social network server 1522 include identifying other users in the messaging system 1500 with whom a particular user has a relationship or who the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0106] The messaging client 1502 may notify the user of the user device 328 or other users related to such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 1502 may provide notifications to participants in a conversation (e.g., a chat session) in the messaging client 1502 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0107] Fig.161600 is a block diagram illustrating a software architecture 1604 that may be installed on any one or more of the devices described herein. The software architecture 1604 is supported by hardware, such as a machine 1602 including a processor 1620, a memory 1626, and an I / O component 1638. In this example, the software architecture 1604 may be conceptualized as a stack of layers, wherein each layer provides specific functionality. The software architecture 1604 includes layers such as an operating system 1612, a library 1608, a framework 1610, and an application 1606. In operation, the application 1606 invokes an API call 1650 through the software stack and receives a message 1652 in response to the API call 1650.

[0108] The operating system 1612 manages hardware resources and provides public services. The operating system 1612 includes, for example, a kernel 1614, services 1616, and drivers 1622. The kernel 1614 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1614 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1616 can provide other public services to other software layers. Drivers 1622 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1622 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0109] Libraries 1608 provide low-level common infrastructure used by applications 1606. Libraries 1608 may include system libraries 1618 (eg, C standard libraries) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the library 1608 may include an API library 1624, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting graphics content in two dimensions (2D) and three dimensions (3D) on a display, GLMotif for implementing a 3D user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1608 may also include various other libraries 1628 to provide many other APIs to the application 1606.

[0110] The framework 1610 provides a high-level common infrastructure used by the applications 1606. For example, the framework 1610 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 1610 can provide a wide range of other APIs that can be used by the applications 1606, some of which can be specific to a particular operating system or platform.

[0111] In an example, applications 1606 may include a home application 1636, a contacts application 1630, a browser application 1632, a shopping application 1634, a location application 1642, a media application 1644, a messaging application 1646, a game application 1648, and a variety of other applications such as third-party applications 1640. Application 1606 is a program that executes the functions defined in the program. Various programming languages ​​can be used to create one or more of the applications 1606 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 1640 (e.g., provided by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, the third-party application 1640 can call the API call 1650 provided by the operating system 1612 to facilitate the functions described in this article.

[0112] Fig.17 1700 in which instructions 1710 (e.g., software, programs, applications, applet, app, or other executable code) may be executed to cause the machine 1700 to perform any one or more of the methods discussed herein. For example, the instructions 1710 may cause the machine 1700 to perform any one or more of the methods described herein. The instructions 1710 transform a general-purpose, unprogrammed machine 1700 into a specific machine 1700 that is programmed to perform the functions described and illustrated in the manner described. The machine 1700 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1700 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1700 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1710 that specify actions to be taken by the machine 1700, either sequentially or otherwise. In addition, while a single machine 1700 is shown, the term "machine" may also be taken to include a collection of machines that individually or jointly execute instructions 1710 to perform any one or more of the methodologies discussed herein.

[0113] The machine 1700 may include a processor 1702, a memory 1704, and an I / O component 1706, which may be configured to communicate with each other via a bus 1744. In an example, the processor 1702 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1708 that executes instructions 1710 and a processor 1712. The term "processor" is intended to include a multi-core processor, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.17 Multiple processors 1702 are shown, but the machine 1700 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0114] The memory 1704 includes a main memory 1714, a static memory 1716, and a storage unit 1718, which are all accessible by the processor 1702 via the bus 1744. The main memory 1704, the static memory 1716, and the storage unit 1718 store instructions 1710 that implement any one or more of the methods or functions described herein. During execution of the instructions 1710 by the networked system 300, the instructions 1710 may also reside, in whole or in part, within the main memory 1714, within the static memory 1716, within the machine-readable medium 1720, within the storage unit 1718, within one or more of the processors 1702 (e.g., within a cache memory of a processor), or within any suitable combination thereof.

[0115] I / O components 1706 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 1706 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O components 1706 may include Fig.171706. In various examples, the I / O components 1706 may include output components 1728 and input components 1732. The output components 1728 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1732 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide location and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0116] In another example, the I / O component 1706 may include: a biometric component 1734, a motion component 1736, an environment component 1738, or a positioning component 1740, as well as various other components. For example, the biometric component 1734 includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1736 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1738 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1740 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure that can obtain altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0117] A variety of technologies may be used to achieve communication. I / O components 1706 also include communication components 1742 that are operable to couple networked system 300 to network 1722 or device 1724 via coupling 1730 and coupling 1726, respectively. For example, communication components 1742 may include a network interface component or another suitable device that interfaces with network 1722. In other examples, communication components 1742 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Parts (e.g. Low power consumption), Device 1724 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0118] In addition, the communication component 1742 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1742 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Supercode, UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1742, such as a location obtained via Internet Protocol (IP) geolocation, ... Location obtained by signal triangulation, location obtained via detection of NFC beacon signals that can indicate a specific location, etc.

[0119] Various memories (e.g., memory 1704, main memory 1714, static memory 1716, and / or memory of processor 1702) and / or storage unit 1718 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1710) when executed by processor 1702 cause various operations to implement the disclosed examples.

[0120] The instructions 1710 may be transmitted or received via a network interface device (e.g., a network interface component included in the communication component 1742) using a transmission medium and using any of a number of well-known transmission protocols (e.g., the Hypertext Transfer Protocol (HTTP)) over the network 1722. Similarly, the instructions 1710 may be transmitted or received to the device 1724 via a coupling 1726 (e.g., a peer-to-peer coupling) using a transmission medium.

[0121] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0122] A "user device" or "client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other user devices or client devices. A user device or client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0123] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0124] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed to be used with other components and a part of a program that generally performs a specific function of related functions. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (e.g., by software) or permanently configured to perform one or more processors of related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein one or more specific processors are examples of hardware. For example, some operations in the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0125] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0126] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Thus, the term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term "signal media".

[0127] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0128] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0129] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

[0130] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0131] A "user device" or "client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other user devices or client devices. A user device or client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0132] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, including the third partnership project (3GPP) of 3G, fourth wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0133] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed to be used with other components and a part of a program that generally performs a specific function of a related function. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (e.g., by software) or permanently configured to perform one or more processors of related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein one or more specific processors are examples of hardware. For example, some operations in the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0134] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0135] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Thus, the term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term "signal media".

[0136] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0137] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0138] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A computer-implemented method using a head mounted display device including a camera and one or more displays, the method comprising: detecting the presence of an optical code in a field of view of the camera device of the head mounted display device; obtaining product information and pricing information for an item based on information in the optical code, the product information and the pricing information including an item identifier and a price; displaying the item identifier and the price in a first display area of ​​the one or more displays of the head mounted display device; as well as The item identifier and the price are communicated to a shopping cart list in a second display area of ​​the one or more displays of the head mounted display device.

2. The computer-implemented method of claim 1 , wherein: Transmitting the item identifier and the price to the shopping cart list occurs after a delay and without user input.

3. The computer-implemented method of claim 1 , wherein: The head mounted display device includes a touch panel located on a side of the head mounted display device, the method further comprising: displaying a default item quantity for the item in the first display area; and An input is received on the touch panel to change the default item quantity for the item to an updated item quantity.

4. The computer-implemented method of claim 3, further comprising: Based on the updated item quantity being zero, removing the item identifier and the price from the first display area without transferring the item identifier and the price to the second display area.

5. The computer-implemented method of claim 1 , further comprising: comparing the product information for the item with product information in a user profile; determining, based on the comparing, that the item is not preferred; as well as A warning is displayed in the first display area.

6. The computer-implemented method of claim 5, further comprising: Based on determining that the item is not preferred, a default item quantity for the item is set to zero.

7. The computer-implemented method of claim 1 , wherein: The head mounted display device includes a touch panel located on a side of the head mounted display device, the method further comprising: receiving a user selection of an item identifier in the shopping cart list; displaying the item identifier and the price in the first display area; and User input on the touch pad is received to change the item quantity for the item to an updated item quantity.

8. The computer-implemented method of claim 1 , further comprising: Determine store identity; as well as Product availability and pricing information is accessed based on the store identification.

9. The computer-implemented method of claim 8, wherein: The store identification is determined from the group consisting of: scanning a store identification optical code located at the store; a short range RF signal transmitted at the store; and The GPS location of said store.

10. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations for providing a user interface in a head-mounted display device comprising a camera and one or more displays, the operations comprising: detecting the presence of an optical code in a field of view of the camera device of the head mounted display device; obtaining product information and pricing information for an item based on information in the optical code, the product information and the pricing information including an item identifier and a price; displaying the item identifier and the price in a first display area of ​​the one or more displays of the head mounted display device; as well as The item identifier and the price are communicated to a shopping cart list in a second display area of ​​the one or more displays of the head mounted display device.

11. The non-transitory computer-readable storage medium of claim 10, wherein: Transmitting the item identifier and the price to the shopping cart list occurs after a delay and without user input.

12. The non-transitory computer-readable storage medium of claim 10, wherein: The head mounted display device includes a touch panel located on a side of the head mounted display device, the operations further comprising: displaying a default item quantity for the item in the first display area; and An input is received on the touch panel to change the default item quantity for the item to an updated item quantity.

13. The non-transitory computer readable storage medium of claim 10, the operations further comprising: comparing the product information for the item with product information in a user profile; Based on the comparing, determining that the item is not preferred; and A warning is displayed in the first display area.

14. The non-transitory computer-readable storage medium of claim 10, wherein: The operations also include: Determine store identity; and Product availability and pricing information is accessed based on the store identification.

15. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the apparatus to perform operations for providing a user interface in a head mounted display device including a camera and one or more displays, the operations comprising: detecting the presence of an optical code in a field of view of the camera device of the head mounted display device; obtaining product information and pricing information for an item based on information in the optical code, the product information and the pricing information including an item identifier and a price; displaying the item identifier and the price in a first display area of ​​the one or more displays of the head mounted display device; and The item identifier and the price are communicated to a shopping cart list in a second display area of ​​the one or more displays of the head mounted display device.

16. The computing device of claim 15, wherein: Transmitting the item identifier and the price to the shopping cart list occurs after a delay and without user input.

17. The computing device of claim 15, the operations further comprising: displaying a default item quantity for the item in the first display area; as well as An input is received on the touch panel to change the default item quantity for the item to an updated item quantity.

18. The computing device of claim 15, the operations further comprising: comparing the product information for the item with product information in a user profile; Based on the comparing, determining that the item is not preferred; and A warning is displayed in the first display area.

19. The computing device of claim 18, the operations further comprising: Based on determining that the item is not preferred, a default item quantity for the item is set to zero.

20. The computing device of claim 15, the operations further comprising: scanning an optical code including information defining a store logo; transmitting information defining the store identifier to a remote server, and Product availability and pricing information is received based on the store identification.

Images