Guided Retail Experience

Abstract

The present disclosure relates to guided retail experiences. The present disclosure relates to techniques for implementing a retail experience in a computer-generated reality environment. In some embodiments, the techniques include: generating a computer-generated reality environment, including presenting a virtual object representing a product in the computer-generated reality environment; while presenting the virtual object representing the product in the computer-generated reality environment, providing a communication session with a remote user of the product; and while providing the communication session, using information received from the communication session to adjust the presentation of the virtual object representing the product in the computer-generated reality environment.

Description

Technical Field

[0001] The present disclosure relates generally to computer generated reality (CGR) environments, and more particularly to representation of electronic products in CGR environments. Background Art

[0002] Conventional electronic consumer experiences allow users to browse and purchase products online with electronic devices such as laptops or desktop computers, tablet computers or smart phones. Users can access product information, select products and purchase products through websites or applications without interacting with another person. Summary of the invention

[0003] Described herein is a technology for implementing a retail experience in a CGR environment. Online shopping may lack instant feedback about products, answers to questions, suggestions, demonstrations, and the personal connection of a face-to-face shopping experience. Current online consumer applications require users to browse menus or perform search functions to locate products of interest. In addition, users cannot fully interact with the online representation of the product, and therefore cannot truly experience the complete demonstration of the product when in a remote environment. The technology described herein provides real-time communication (via a virtual communication session) with a remote salesperson who can guide users in a shopping experience in a CGR environment. This allows users to stay in a remote environment while having an interactive shopping experience with a salesperson who can provide relevant products, give suggestions based on interactions, and facilitate a complete demonstration of various products and services in a CGR environment. The salesperson can display a product or a collection of products in a CGR environment, and the user can interact with the product to perform a demonstration of the product with or without input from the salesperson. Even without input from the user, the salesperson can manipulate the product to show features and demonstrate the ability of the product. The technology disclosed herein provides users with a retail experience in a remote environment such as the user's house but with the benefits and capabilities of a face-to-face shopping experience.

[0004] In some embodiments, technology for implementing a retail experience in a CGR environment includes: generating a CGR environment at a first electronic device, including presenting a virtual object representing a product in the CGR environment; providing a communication session with a remote user while presenting the virtual object representing the product in the CGR environment; and while providing the communication session, using information received from the communication session to adjust the presentation of the virtual object representing the product in the CGR environment.

[0005] In some embodiments, the communication session is provided in response to receiving a request to initiate a communication session.

[0006] In some embodiments, presenting the virtual object representing the product includes superimposing the virtual object on a physical display of a physical device in the CGR environment.

[0007] In some embodiments, the technology also includes: while providing the communication session, detecting an input; and in response to detecting the input, modifying a virtual object representing the product in the CGR environment based on the detected input.

[0008] In some embodiments, the input corresponds to a gesture.

[0009] In some embodiments, modifying virtual objects representing products in the CGR environment based on the detected input includes modifying the appearance of one or more virtual objects based on at least one of a magnitude and a direction of the gesture.

[0010] In some embodiments, using information received from the communication session to adjust the presentation of a virtual object representing a product in the CGR environment includes presenting a different virtual object.

[0011] In some embodiments, using information received from the communication session to adjust the presentation of a virtual object representing a product in the CGR environment includes ceasing to present the virtual object.

[0012] In some embodiments, the product is a physical product that is configured to perform a set of operations in response to detecting a first set of inputs to the physical product; and the virtual object representing the product is a virtual representation of the physical product that is configured to perform a set of operations in response to detecting the first set of inputs to the virtual representation of the physical product.

[0013] In some embodiments, generating the CGR environment includes presenting a virtual representation of the salesperson in the CGR environment.

[0014] In some embodiments, the CGR environment is an augmented reality environment. In some embodiments, the technology also includes: detecting a second input; and in response to detecting the second input, transitioning the CGR environment to a virtual reality environment while continuing to provide the communication session.

[0015] In some embodiments, a CGR environment is generated in response to detecting input directed to an object associated with an application presented using an electronic device.

[0016] In some embodiments, the technique also includes generating a data stream to be received by a second electronic device associated with the remote user, and the data stream includes a representation of at least a portion of the CGR environment.

[0017] In some embodiments, a system, electronic device, or computer-readable storage medium (transitory or non-transitory) provides the described techniques. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In the following description, reference is made to the accompanying drawings which form a part of the description and illustrate several embodiments. It should be understood that other embodiments may also be utilized, and structural and operational changes may be made without departing from the scope of the present disclosure. The same reference symbols are used in different drawings to represent similar or identical items.

[0019] Figure 1A to Figure 1B Depicted are exemplary systems for use in various computer-generated reality techniques.

[0020] FIG. 2A to FIG. 2H Devices showing a retail experience implemented in a CGR environment are shown.

[0021] Figure 3 A flow chart depicts an exemplary technique for implementing a retail experience in a CGR environment. DETAILED DESCRIPTION

[0022] Various examples of electronic systems and techniques for using such systems in connection with various computer-generated reality techniques are described.

[0023] The physical environment (or real environment) refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical items (or physical objects or real objects), such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0024] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that a person perceives and / or interacts with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0025] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object, which creates a 3D or spatial audio environment that provides the perception of a point audio source in a 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0026] Examples of CGR include virtual reality and mixed reality.

[0027] A virtual reality (VR) environment (virtual environment) refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment, and / or through a simulation of a subset of the person's physical movements within the computer-generated environment.

[0028] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, an MR environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0029] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0030] Examples of MR include augmented reality and augmented virtuality.

[0031] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0032] An AR environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not true version of the original captured image. For another example, a representation of a physical environment may be transformed by graphically eliminating or blurring a portion thereof.

[0033] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more features of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the position of the sun in the physical environment.

[0034] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one example, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0035] Figure 1A and Figure 1B An exemplary system 100 for use in various computer-generated reality techniques is depicted.

[0036] In some examples, such as Figure 1A As shown, system 100 includes device 100a. Device 100a includes various components, such as one or more processors 102, one or more RF circuits 104, one or more memories 106, one or more image sensors 108, one or more orientation sensors 110, one or more microphones 112, one or more position sensors 116, one or more speakers 118, one or more displays 120, and one or more touch-sensitive surfaces 122. These components optionally communicate via one or more communication buses 150 of device 100a.

[0037] In some examples, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, a mobile device, or a laptop computer), and other elements of system 100 are implemented in a head mounted display (HMD) device designed to be worn by a user, wherein the HMD device communicates with the base station device. In some examples, device 100a is implemented in either the base station device or the HMD device.

[0038] like Figure 1B As shown, in some examples, the system 100 includes two (or more) devices in communication, such as through a wired connection or a wireless connection. The first device 100b (e.g., a base station device) includes one or more processors 102, one or more RF circuits 104, and one or more memories 106. These components optionally communicate through one or more communication buses 150 of the device 100b. The second device 100c (e.g., a head-mounted device) includes various components, such as one or more processors 102, one or more RF circuits 104, one or more memories 106, one or more image sensors 108, one or more orientation sensors 110, one or more microphones 112, one or more position sensors 116, one or more speakers 118, one or more displays 120, and one or more touch-sensitive surfaces 122. These components optionally communicate through one or more communication buses 150 of the device 100c.

[0039] In some examples, system 100 is a mobile device. In some examples, system 100 is a head mounted display (HMD) device. In some examples, device 100 is a wearable HUD device.

[0040] System 100 includes one or more processors 102 and one or more memories 106. One or more processors 102 include one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 102 to perform the techniques described below.

[0041] System 100 includes one or more RF circuits 104. One or more RF circuits 104 optionally include circuits for communicating with electronic devices, networks (such as the Internet, an intranet), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). One or more RF circuits 104 optionally include circuits for using near field communications and / or short range communications (such as ) circuit for communication.

[0042] System 100 includes one or more displays 120. In some examples, display 120 includes a first display (e.g., a left eye display panel) and a second display (e.g., a right eye display panel), each display being used to display an image to a corresponding eye of a user. Corresponding images are displayed simultaneously on the first display and the second display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby generating a parallax effect that provides the user with a stereoscopic effect of the object on the display. In some examples, display 120 includes a single display. For each eye of the user, the corresponding image is simultaneously displayed on a first area and a second area of ​​the single display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby generating a parallax effect that provides the user with a stereoscopic effect of the object on the single display.

[0043] In some examples, system 100 includes a touch-sensitive surface 122 for receiving user inputs, such as tap inputs and swipe inputs. In some examples, one or more displays 120 and one or more touch-sensitive surfaces 122 form one or more touch-sensitive displays.

[0044] The system 100 includes one or more image sensors 108. The image sensor 108 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real environment. The image sensor also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real environment. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the real environment. The image sensor 108 also optionally includes one or more event cameras configured to capture the movement of physical objects in the real environment. The image sensor 108 also optionally includes one or more depth sensors configured to detect the distance of the physical object from the system 100. In some examples, the system 100 uses a combination of CCD sensors, event cameras, and depth sensors to detect the physical environment around the system 100. In some examples, the one or more image sensors 108 include a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical objects in the real environment from two different perspectives. In some examples, the system 100 uses one or more image sensors 108 to receive user input, such as gestures. In some examples, the system 100 uses the image sensor 108 to detect the position and orientation of the system 100 and / or the display 120 in the real environment. For example, the system 100 uses the image sensor 108 to track the position and orientation of the display 120 relative to one or more fixed objects in the real environment.

[0045] In some examples, system 100 includes microphone 112. System 100 uses microphone 112 to detect sounds from the user and / or the user's real environment. In some examples, microphone 112 includes a microphone array (including multiple microphones), which are optionally operated in series to identify ambient noise or locate a sound source in the space of the real environment.

[0046] The system 100 includes one or more orientation sensors 110 for detecting the orientation and / or movement of the system 100 and / or one or more displays 120. For example, the system 100 uses the orientation sensors 110 to track changes in the position and / or orientation of the system 100 and / or the display 120, such as with respect to physical objects in a real environment. The one or more orientation sensors 110 optionally include one or more gyroscopes and / or one or more accelerometers.

[0047] Reference now FIG. 2A to FIG. 2H , describes exemplary techniques for implementing a retail experience in a CGR environment.

[0048] Figure 2A Device 200 is shown having a display 202. Device 200 is similar to device 100a, and display 202 is similar to display 120. Figure 2A In the illustrated embodiment, device 200 represents a head mounted device, but in some embodiments, device 200 may be a smart phone, tablet computer, desktop computer, laptop computer, or other electronic system or device with a display (e.g., Figure 1A to Figure 1B In some embodiments, device 200 communicates with a mouse, keyboard, camera, controller, or other peripheral device that receives input.

[0049] exist Figure 2A 2, display 202 shows a representation of a CGR environment 204, which includes physical objects such as a laptop 206 and a television 208. In some embodiments, display 202 is transparent, and the physical objects are viewed through display 202. In some embodiments, display 202 is opaque, and the physical objects are rendered on display 202. For example, the physical environment is displayed on display 202 as a representation of the physical environment, which includes a physical laptop and a television.

[0050] exist Figure 2B 2, the display 202 displays a prompt 210 in the CGR environment 204. The prompt 210 is a virtual object that provides the user with an option to initiate a retail experience in the CGR environment 204. For example, the user can gesture to accept or decline the option to contact the retailer by waving their hand in one direction or another. In some embodiments, the display of the prompt 210 is initiated by the user. For example, the user can view a website using the laptop 206 and select a link from the website to initiate a retail experience, which will initiate the display of the prompt 210. In some embodiments, the prompt 210 is automatically displayed by the device 200. For example, the device 200 detects the laptop 206 in the CGR environment 204, determines the retailer that sells the laptop 206, and initiates the prompt 210 to contact the retailer that sells the laptop 206. In some embodiments, the prompt 210 is displayed by a virtual assistant or other artificial intelligence that communicates with the device 200. For example, the user can tell the virtual assistant that the user is interested in purchasing a computer similar to the laptop 206. In response to the voice input from the user, the virtual assistant may initiate contact with the retailer, including, for example, displaying prompt 210 as a way to confirm the user's instruction to contact the retailer.

[0051] In response to detecting user input for contacting the retailer (e.g., a gesture for confirming an instruction to contact the retailer), the device 200 initiates a communication session with the retailer to enable a retail experience between the user and the retailer in the CGR environment 204. In some embodiments, the communication session is implemented such that representations of contacts associated with the retailer (e.g., a salesperson) are displayed in the CGR environment 204, and the user is able to interact with the retailer by talking to the salesperson and manipulating virtual objects in the CGR environment 204. Figures 2C to 2H An example of such an embodiment is shown in , where the device 200 displays an avatar 215 in the CGR environment 204. In the illustrated embodiment, the avatar 215 is a virtual representation of a salesperson associated with a retailer that is contacted using the device 200.

[0052] In some embodiments, the communication session includes a two-way audio feed so that the user and the salesperson can talk to each other. For example, the device 200 may include a microphone (e.g., microphone 112) for receiving audio input from the user and one or more speakers (e.g., speaker 118) for conveying audio from the salesperson. In some embodiments, the communication session includes a limited two-way shared video feed, in which the user can view the entire CGR environment 204, but the salesperson can only view a limited portion of the CGR environment 204. For example, in such embodiments, the device 200 transmits a data stream including a visual representation of a virtual object but not a physical object in the CGR environment 204 to a retailer (in some embodiments, the data stream also includes an audio feed). Therefore, the salesperson receiving the data stream cannot view the physical objects of the CGR environment 204 and can only view and interact with the virtual objects displayed in the CGR environment 204. This limiting shared video feed allows the device 200 to facilitate an effective retail experience while also retaining the privacy of the user. The retail experience is effective because both the user and the salesperson can view and interact with the products displayed as virtual objects in the CGR environment 204. The user's privacy is preserved because the salesperson cannot view the physical environment, which may be a private environment, such as, for example, the user's home.

[0053] The salesperson is able to communicate with the user and manipulate virtual objects displayed in the CGR environment 204. Thus, input provided by the salesperson can affect changes in the CGR environment 204 experienced by both the user and the salesperson. For example, the salesperson can manipulate the position of a virtual object, and the user can see the manipulation of the virtual object in the CGR environment 204 via the device 200. Conversely, input provided by the user can affect changes in what is experienced by both the user and the salesperson (e.g., when the user manipulates a virtual object, the salesperson can see the virtual object being manipulated). In some embodiments, the avatar 215 represents the computer-generated position of the salesperson in the CGR environment 204, and changes made by the salesperson to the virtual object are shown in the CGR environment 204 as being performed by the avatar 215. In some embodiments, the salesperson participates in the communication session using a device similar to the device 200. Although in some embodiments, the salesperson is able to participate in the communication session using other devices that can implement the above-mentioned aspects of the communication session.

[0054] Figure 2C to Figure 2G Various exemplary embodiments of a device 200 showing a retail experience implemented in a CGR environment 204 are shown.

[0055] exist Figure 2C , device 200 displays avatar 215 along with laptop 206 and television 208 in CGR environment 204. In this embodiment, the data stream communicated using device 200 does not include visual representations of laptop 206 or television 208 (or other physical objects in the physical environment). Thus, the salesperson is unable to view these and any other physical objects of CGR environment 204, while the user is able to view the contents of display 202, including avatar 215, laptop 206, television 208, and other physical objects in the physical environment.

[0056] In some embodiments, the device 200 can communicate the details of the physical object to the salesperson without communicating a visual representation of the physical object. For example, the device 200 can communicate information such as the brand and model of the laptop 206. In some embodiments, the device 200 can recognize the laptop 206 and retrieve (e.g., from a database or by accessing information from an information network) the details of the laptop to communicate to the salesperson. In some embodiments, the user provides the details of the laptop 206 when communicating with the salesperson.

[0057] exist Figure 2D, the device 200 displays the first software 211 on the display screen 206-1 of the laptop 206. In some embodiments, the display of the first software 211 is initiated by a user. In some embodiments, the display of the first software 211 is initiated by a salesperson to, for example, provide a demonstration of software that can be operated using the laptop 206. For example, the salesperson can control various aspects of the retail experience to initiate the display of different products displayed as virtual objects in the CGR environment 204. Although the laptop 206 is not operating the first software 211 in the physical environment, the device 200 displays the first software 211 as a virtual object that appears on the display screen 206-1 to provide the appearance of the first software 211 operating on the laptop 206 in the CGR environment 204. In some embodiments, the first software 211 represents fully interactive software that responds to input provided using the laptop 206. For example, a user can interact with the laptop 206 in the CGR environment 204 (e.g., by typing on the keyboard 206-2). The device 200 detects the user's input on the keyboard 206-2 (e.g., by identifying the keyboard keys that the user is touching), and uses the detected input to interact with the first software 211. In this way, the user can interact with the physical laptop 206 in the CGR environment 204 to fully demonstrate the capabilities of the first software 211 operating on the laptop 206.

[0058] exist Figure 2E, the device 200 displays the second software 212 as a virtual object appearing on the display screen 206-1 in the CGR environment 204. The device 200 is capable of providing a complete demonstration of the second software 212 in a manner similar to that described above with respect to the first software 211. In some embodiments, the device 200 replaces the first software 211 with the second software 212 in response to receiving input from the salesperson (e.g., in a manner similar to that discussed above). In some embodiments, the device 200 replaces the first software 211 with the second software 212 in response to detecting input from the user. For example, in some embodiments, the device 200 may detect a gesture from the user (with or without audio or other input provided from the user) that corresponds to a command for displaying a different virtual object in the CGR environment 204. In some embodiments, the gesture has a direction and / or magnitude component that can be detected by the device 200 to affect the input provided from the user. Using software applications (e.g., first software 211, second software 212) as an example, device 200 may cycle between different software applications displayed in CGR environment 204 in response to detecting a swipe gesture of a user's hand. If device 200 detects that the user makes a short swipe gesture in a first direction, device 200 transitions from the first software application in the virtual arrangement of software applications to the next software application in the arrangement (e.g., from application A to application B). If device 200 detects that the user makes a long swipe gesture in the first direction, device 200 transitions to a different software application with a larger displacement in the virtual arrangement of software applications (e.g., from application A to application E). When device 200 detects a gesture in the opposite direction, device 200 transitions applications in a similar manner but in the opposite direction (e.g., from application B to application A, or from application E to application A, depending on the detected magnitude of the gesture).

[0059] Figure 2F and Figure 2G An embodiment is shown in which a salesperson introduces a display of a product 220 and a user interacts with the product in the CGR environment 204 as part of a demonstration of the product during a retail experience. In some embodiments, a user may introduce a display of a product 220 in the CGR environment 204.

[0060] exist Figure 2F2, device 200 displays avatar 215 posing with product 220, which is displayed as a virtual object appearing on television 208. In some embodiments, the appearance of television 208 is modified so that it appears to be a product display case housing product 220. It should be understood that product 220 can be displayed in other locations of CGR environment 204. For example, the product can be displayed on other physical objects such as table 218 (e.g., adjacent laptop 206).

[0061] Product 220 is a virtual object displayed in CGR environment 204. A user can interact with product 220 to experience a demonstration of the product in CGR environment 204. Figure 2G In the illustrated embodiment, the user retrieves a smartphone 220-1 from a product 220 displayed on a television 208, and the device 200 shows that the user's hand 222 holds the retrieved smartphone 220-1. The smartphone 220-1 is a virtual smartphone that is a fully functional virtual representation of a physical smartphone. The smartphone 220-1 is shown in a CGR environment 204 with a virtual display 224 that displays virtual application icons 226. In the CGR environment 204, the device 200 displays the smartphone 220-1, which behaves as if the virtual object is a real physical object on which the virtual object is modeled. Therefore, when the device 200 detects an input on the smartphone 220-1, the device 200 displays that the smartphone 220-1 responds in the same manner as a physical smartphone would respond to the same input in a physical environment. In this way, the device 200 enables the user to experience a complete and accurate demonstration of the smartphone by interacting with the smartphone 220-1 in the CGR environment 204. It should be understood that the user can interact in a similar manner with other products represented in the CGR environment 204. For example, the device 200 can enable the user to interact with a virtual representation of a laptop, monitor, tablet, or any other device that can be represented in the CGR environment 204.

[0062] In some embodiments, the device 200 enables the user to modify the virtual object in other ways. For example, the device 200 may display annotations, highlights, or other visual modifications in the CGR environment 204 so that the user can further interact with the salesperson. This can enhance the retail experience by allowing the user to indicate specific features or areas of interest in the CGR environment 204.

[0063] In some embodiments, the device 200 can change the CGR experience displayed to the user. For example, in some embodiments such as Figure 2GIn the illustrated embodiment, the CGR environment 204 is an AR environment that includes both physical objects (e.g., laptop 206, television 208, table 218) and virtual objects (e.g., product 220, smartphone 220-1, avatar 215). In response to input (e.g., input provided by a salesperson or user), the device 200 may move the CGR environment 204 from one physical object to another. Figure 2G The AR environment shown is transformed into a VR environment, such as Figure 2H As shown. Figure 2H , the CGR environment 204 is now a VR environment including a virtual environment with an avatar 215, a virtual table 228, and a virtual product 230. In the VR environment of the CGR environment 204, the device 200 no longer displays physical objects, such as the laptop 206, the television 208, and the table 218. Figure 2H In the illustrated embodiment, the VR environment is a virtual retail store displaying virtual products 230. However, it should be understood that other virtual environments may be displayed in the VR environment, including representations of real locations (e.g., the Great Wall of China) or fictitious virtual locations (e.g., a computer-generated concert).

[0064] In some embodiments, device 200 allows a user to initiate a retail experience without involving a salesperson (e.g., avatar 215). For example, device 200 may display one or more virtual objects, such as a display device, in response to a user's request to display various devices. Figure 2F For example, a user may interact with a website or application to initiate a retail experience and select one or more products to view in the CGR environment 204. The products may be displayed as virtual objects in the CGR environment 204, and the user may interact with the virtual products in a manner similar to that discussed above.

[0065] In some embodiments, the salesperson (e.g., avatar 215) is a virtual salesperson, such as a virtual assistant or a form of artificial intelligence for interacting with the user in the CGR environment 204. In such embodiments, the device 200 can communicate with the virtual salesperson (or a network providing a virtual salesperson) to provide information about the user and / or other devices in communication with the device 200. For example, in such embodiments, the device 200 can access data that informs the device 200 that the user is interested in a particular topic such as, for example, music. This information can be used by the device 200 to influence the user's retail experience in the CGR environment 204. For example, the device 200 can display or suggest products with music applications that the user may be interested in. As another example, the CGR environment 204 can be displayed as a VR environment in which the user can experience using virtual products (e.g., speakers, audio / video software, etc.) to simulate performing as a disc jockey in front of a large number of virtual users (e.g., computer-generated people in the VR environment, or in some embodiments, real people who experience the VR environment with the user).

[0066] It should be appreciated that the above techniques may be implemented to provide other experiences in the CGR environment 204. For example, instead of implementing a retail experience for purchasing a product, the device 200 may implement an experience in the CGR environment 204 that allows the user to troubleshoot a device in the CGR environment 204. For example, a user may implement the device 200 to initiate contact with a retailer to troubleshoot a laptop 206. As another example, the device 200 may implement a classroom environment in which a user interacts with another user (e.g., a teacher) to view and experience a product in the CGR environment 204 in order to learn how to use and troubleshoot the product displayed as a virtual object in the CGR environment 204.

[0067] Now go to Figure 3 , depicts a flow chart of an exemplary technique 300 for implementing a retail experience in a CGR environment. Technique 300 may be performed using a user device (e.g., 100a or 200). Figure 3 The blocks of the technique 300 are depicted in a particular order, but these blocks may be performed in other orders. In addition, one or more blocks of the technique 300 may be optional and / or additional blocks may be performed.

[0068] At block 302, a device (e.g., 100a, 200) generates a computer-generated reality environment (e.g., CGR environment 204) (e.g., a mixed reality environment; e.g., an augmented reality environment) including presenting a virtual object (e.g., first software 211; smart phone 220-1) representing a product (e.g., a software product; a smart phone product) in the CGR environment (e.g., a representation of a product sold by a retail provider in a product showroom; e.g., a physical object; e.g., a virtual object). In some embodiments, presenting the virtual object involves presenting a virtual object on a screen such as Figure 1A and Figure 1B Display 120 or FIG. 2A to FIG. 2H The virtual object is displayed on the display 202 in.

[0069] In some embodiments, the product is a physical product (e.g., a smart phone / tablet) that is configured to perform a set of operations in response to detecting a first set of inputs to the physical product (e.g., the physical smart phone / tablet is configured to open and display a home page with applications on a physical screen in response to detecting inputs on a physical power button located on the physical smart phone / tablet). In some embodiments, the virtual object representing the product (e.g., smart phone 220-1) is a virtual representation of a physical product (e.g., a virtual smart phone or virtual tablet) that is configured to perform a set of operations in response to detecting a first set of inputs to the virtual representation of the physical product (e.g., the virtual smart phone / tablet is configured to open and display a home page with applications (e.g., 226) on a virtual screen (e.g., 224) in response to inputs on a power button that is presented as appearing on the virtual smart phone / tablet).

[0070] In some embodiments, generating a computer-generated reality environment (e.g., CGR environment 204) includes presenting a virtual representation (e.g., salesperson 215) (e.g., a virtual avatar) of a remote user (e.g., a salesperson) in the computer-generated reality environment. In some embodiments, the virtual representation of the remote user is presented in the computer-generated reality environment as a separate entity (e.g., another user) present in the computer-generated reality environment, and the virtual representation of the remote user can guide the user through a shopping experience in the computer-generated reality environment.

[0071] In some embodiments, a computer-generated reality environment (e.g., CGR environment 204) is generated in response to detecting input for an object (e.g., a link, an affordance, etc.) associated with an application (e.g., a shopping application, a news application, a web browser application, etc.) presented using an electronic device.

[0072] At box 304, while presenting a virtual object (e.g., 211, 220-1) representing a product in a computer-generated reality environment (e.g., a communication session in which a representation of the salesperson is optionally presented in the computer-generated reality environment), the device (e.g., 100a, 200) provides a communication session (e.g., a virtual communication session) with a remote user (e.g., a salesperson of the product) (e.g., a second user; e.g., a representative of a retail provider; e.g., an automated or virtual assistant). In some embodiments, the communication session is associated with a virtual retail experience. In some embodiments, the communication session includes a two-way audio feed between the electronic device and the salesperson. In some embodiments, the communication session includes a limited two-way shared video feed in which the user of the electronic device is able to view the entire computer-generated reality environment, including using visual input provided by a second electronic device associated with the salesperson, and the salesperson is able to view limited content of the computer-generated reality environment.

[0073] In some embodiments, the communication session is provided in response to receiving a request to initiate a communication session (e.g., interacting with prompt 210 to initiate a communication session) (e.g., initiated by a user; e.g., initiated by a salesperson (e.g., a representative of a retail provider)).

[0074] In some embodiments, presenting (e.g., displaying) a virtual object representing a product (e.g., first software 211) includes superimposing the virtual object (e.g., one or more virtual representations of a software application configured to operate with the physical object) on a physical display (e.g., display 206-1) (or a representation thereof) of a physical device (e.g., laptop 206) in a computer-generated reality environment. In some embodiments, the virtual object is configured to be modified in response to information received from the communication session (e.g., a modification request initiated by a second electronic device). In some embodiments, the information from the communication session includes a request corresponding to switching from presenting the first virtual object to presenting the second virtual object (e.g., from Figure 2D The first software in 211 switches to Figure 2E For example, if the physical device is a laptop (e.g., 206) and one of the virtual objects is a first software application (e.g., 211) configured to operate on the laptop (e.g., the software application is rendered (e.g., overlaid) to be presented on a display screen of the laptop in the computer-generated real environment), then in response to receiving the modification request, the electronic device switches from presenting the first software application on the display screen of the laptop to presenting a different software application (e.g., the second software 212) on the display screen of the laptop.

[0075] At box 306, when providing a communication session, the device (e.g., 100a, 200) uses information received from the communication session (e.g., by a second (e.g., remote) electronic device) to adjust the presentation of a virtual object (e.g., 211, 220-1) representing a product in a computer-generated reality environment. For example, in some embodiments, the information received from the communication session is a modification request initiated by a second electronic device associated with a salesperson. In some embodiments, the modification request corresponds to a request for a first electronic device (e.g., device 200) to modify at least one characteristic of the computer-generated reality environment (e.g., to modify or present one or more virtual objects in the computer-generated reality environment). In some embodiments, in response to receiving the modification request, the device modifies the appearance of the computer-generated reality environment based on the modification request initiated by the second electronic device. In some embodiments, modifying the appearance of the computer-generated reality environment based on the modification request includes presenting a modification to the object (e.g., changing the orientation / posture of the virtual object, adding a virtual object to the environment, removing a virtual object from the environment). In some embodiments, modifying the appearance of the computer-generated reality environment based on the modification request includes presenting a virtual object associated with an object (e.g., a physical object) in the computer-generated reality environment.

[0076] In some embodiments, using information received from the communication session to adjust the presentation of virtual objects representing products in the computer-generated real environment includes presenting different virtual objects (e.g., the second software 212) (e.g., virtual representations of different products in a product library; e.g., highlights or annotations associated with the objects).

[0077] In some embodiments, using information received from the communication session to adjust the presentation of a virtual object representing a product in the computer-generated reality environment includes stopping presentation of the virtual object (e.g., the first software 211 no longer presents the virtual object in the computer-generated reality environment). Figure 2E ) (e.g., stopping presenting an object (e.g., when the object is a virtual object); e.g., stopping presenting virtual representations of different products in a product library).

[0078] In some embodiments, while providing a communication session, the device detects input (e.g., input provided by a user), and in response to detecting the input, modifies a virtual object representing a product in a computer-generated real-world environment based on the detected input (e.g., modifies the appearance of the object (e.g., virtual object) based on the detected input). In some embodiments, the input corresponds to a posture (e.g., a hand gesture).

[0079] In some embodiments, modifying virtual objects representing products in a computer-generated real-world environment based on detected input includes modifying the appearance of one or more virtual objects based on at least one of the magnitude and direction of a gesture (e.g., switching between a single iteration of items (e.g., presented applications) when the gesture is small, or skipping between groups of items when the gesture is large; for example, moving forward in an order of items (e.g., applications) when the gesture is in a first direction, or moving backward in the order when the gesture is in a second (opposite) direction).

[0080] In some embodiments, the computer-generated reality environment is an augmented reality environment (e.g., an environment with virtual and physical objects), and the technology also includes detecting a second input (e.g., input from a user; e.g., input from a salesperson), and in response to detecting the second input, transitioning the computer-generated reality environment to a virtual environment (e.g., a completely virtual environment without physical objects) while continuing to provide the communication session (e.g., as described in the example of FIG. 1 ). Figure 2H shown).

[0081] In some embodiments, generating the computer-generated reality environment also includes generating a data stream to be received by a second electronic device associated with the salesperson (e.g., the electronic device generates (e.g., via a communication network) a data stream that is transmitted to an electronic device associated with the salesperson). In some embodiments, the data stream includes a representation of at least a portion of the computer-generated reality environment (e.g., 211, 212, 220). In some embodiments, the data stream includes a virtual object representing a product. In some embodiments, the data stream includes only a virtual object representing a product and does not include a representation of the computer-generated reality environment (e.g., 206, 208, 218), such that the salesperson can view a representation of the product but cannot view other portions of the computer-generated reality environment.

[0082] Executable instructions for performing features of the technique 300 described above are, optionally, included in a transitory or non-transitory computer-readable storage medium (e.g., memory 106) or other computer program product configured to be executed by one or more processors (e.g., processor 102).

[0083] The foregoing descriptions of specific embodiments have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the scope of the claims to the precise forms disclosed, and it is to be understood that many modifications and variations are possible in light of the above teachings.

[0084] Aspects of the above technology contemplate the possibility of collecting and using personal information to enable retail experiences in computer-generated reality environments. Such information should be collected with the user's informed consent.

[0085] Entities processing such personal information will adhere to established privacy practices and / or privacy policies (e.g., policies certified by a third party) that (1) are generally recognized as meeting or exceeding industry or government requirements, (2) are accessible to users, (3) are updated as necessary, and (4) comply with applicable law. Entities processing such personal information will use such information for reasonable and lawful purposes and will not share or sell it outside of those lawful purposes.

[0086] However, users may selectively limit access / use of personal information. For example, users may choose to participate or not participate in the collection of their personal information. Furthermore, while aspects of the above-described techniques contemplate the use of personal information, aspects of these techniques may be implemented without the need or use of personal information. For example, if location information, user names, and / or addresses are collected, they may be generalized and / or masked so that they do not uniquely identify an individual.

Claims

1. An electronic device, comprising: monitor; one or more processors; and A memory storing one or more programs configured to be executed by one or more processors, the one or more programs including instructions for: Generate computer-generated environments including presenting in the computer-generated environment a view of a physical environment in which the electronic device is located, wherein the physical environment includes physical objects having physical displays; and presenting a first virtual object representing a first product in the computer-generated environment, wherein the first virtual object is presented superimposed on the physical display of the physical object; providing a communication session with a remote user while presenting the first virtual object representing the first product and the view of the physical environment in the computer-generated environment, wherein providing the communication session with the remote user includes transmitting information corresponding to the physical object and information corresponding to the first virtual object representing the first product without transmitting the view of the physical environment, wherein the information corresponding to the physical object includes product information for the physical object; and While providing the communication session, using information received from the communication session, replacing the presentation of the first virtual object representing the first product in the computer-generated environment with a presentation of a second virtual object representing a second product, wherein the second virtual object is presented superimposed on the physical display of the physical object. 2 . The electronic device of claim 1 , wherein the communication session is provided in response to receiving a request to initiate the communication session.

3. The electronic device according to claim 1, wherein the one or more programs further comprise instructions for: While providing the communication session, detecting an input; and In response to detecting the input, the first virtual object representing the first product in the computer-generated environment is modified based on the detected input.

4. The electronic device according to claim 3, wherein: The input corresponds to a gesture, and Modifying the first virtual object representing the first product in the computer-generated environment based on the detected input includes modifying the appearance of one or more additional virtual objects based on at least one of a magnitude and a direction of the gesture.

5. The electronic device according to claim 1, wherein: The first product is a physical product configured to perform a set of operations in response to detecting a first set of inputs to the physical product; and The first virtual object representing the first product is a virtual representation of the physical product, the virtual representation being configured to perform the set of operations in response to detecting the first set of inputs directed to the virtual representation of the physical product. 6 . The electronic device of claim 1 , wherein generating the computer-generated environment comprises presenting a virtual representation of the remote user in the computer-generated environment.

7. The electronic device of claim 1, wherein the computer-generated environment is an augmented reality environment, and wherein the one or more programs further comprise instructions for: detecting a second input; and In response to detecting the second input, the computer-generated environment is transitioned to a virtual reality environment while continuing to provide the communication session. 8 . The electronic device of claim 1 , wherein the computer-generated environment is generated in response to detecting an input to an object associated with an application presented using the electronic device.

9. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs comprising instructions for performing the following operations: Generate a computer-generated environment, including: presenting in the computer-generated environment a view of a physical environment in which the electronic device is located, wherein the physical environment includes a physical object having a physical display; as well as presenting a first virtual object representing a first product in the computer-generated environment, wherein the first virtual object is presented superimposed on the physical display of the physical object; providing a communication session with a remote user while presenting the first virtual object representing the first product and the view of the physical environment in the computer-generated environment, wherein providing the communication session with the remote user includes transmitting information corresponding to the physical object and information corresponding to the first virtual object representing the first product without transmitting the view of the physical environment, wherein the information corresponding to the physical object includes product information for the physical object; and While providing the communication session, using information received from the communication session, replacing the presentation of the first virtual object representing the first product in the computer-generated environment with a presentation of a second virtual object representing a second product, wherein the second virtual object is presented superimposed on the physical display of the physical object.

10. The non-transitory computer readable storage medium of claim 9, wherein the one or more programs further comprise instructions for: While providing the communication session, detecting an input; and In response to detecting the input, the first virtual object representing the first product in the computer-generated environment is modified based on the detected input.

11. The non-transitory computer readable storage medium of claim 10, wherein: The input corresponds to a gesture, and Modifying the first virtual object representing the first product in the computer-generated environment based on the detected input includes modifying the appearance of one or more additional virtual objects based on at least one of a magnitude and a direction of the gesture.

12. The non-transitory computer-readable storage medium of claim 9, wherein: The first product is a physical product configured to perform a set of operations in response to detecting a first set of inputs to the physical product; and The first virtual object representing the first product is a virtual representation of the physical product, the virtual representation being configured to perform the set of operations in response to detecting the first set of inputs directed to the virtual representation of the physical product.

13. The non-transitory computer-readable storage medium of claim 9, wherein the computer-generated environment is an augmented reality environment, the one or more programs further comprising instructions for: detecting a second input; and In response to detecting the second input, the computer-generated environment is transitioned to a virtual reality environment while continuing to provide the communication session.

14. The non-transitory computer-readable storage medium of claim 9, wherein the communication session is provided in response to receiving a request to initiate the communication session.

15. The non-transitory computer-readable storage medium of claim 9, wherein generating the computer-generated environment comprises presenting a virtual representation of the remote user in the computer-generated environment.

16. The non-transitory computer-readable storage medium of claim 9, wherein the computer-generated environment is generated in response to detecting an input to an object associated with an application presented using the electronic device.

17. A method comprising: At Electronic Equipment: Generate a computer-generated environment, including: presenting in the computer-generated environment a view of a physical environment in which the electronic device is located, wherein the physical environment includes a physical object having a physical display; presenting a first virtual object representing a first product in the computer-generated environment, wherein the first virtual object is presented superimposed on the physical display of the physical object; providing a communication session with a remote user while presenting the first virtual object representing the first product and the view of the physical environment in the computer-generated environment, wherein providing the communication session with the remote user includes transmitting information corresponding to the physical object and information corresponding to the first virtual object representing the first product without transmitting the view of the physical environment, wherein the information corresponding to the physical object includes product information for the physical object; and While providing the communication session, using information received from the communication session, replacing the presentation of the first virtual object representing the first product in the computer-generated environment with a presentation of a second virtual object representing a second product, wherein the second virtual object is presented superimposed on the physical display of the physical object.

18. The method according to claim 17, further comprising: while providing the communication session, detecting an input; as well as In response to detecting the input, the virtual object representing the product in the computer-generated environment is modified based on the detected input.

19. The method of claim 18, wherein: The input corresponds to a gesture, and Modifying the first virtual object representing the first product in the computer-generated environment based on the detected input includes modifying the appearance of one or more additional virtual objects based on at least one of a magnitude and a direction of the gesture.

20. The method of claim 17, wherein: The first product is a physical product configured to perform a set of operations in response to detecting a first set of inputs to the physical product; and The first virtual object representing the first product is a virtual representation of the physical product, the virtual representation being configured to perform the set of operations in response to detecting the first set of inputs directed to the virtual representation of the physical product.

21. The method of claim 17, wherein the computer-generated environment is an augmented reality environment, the method further comprising: detecting a second input; as well as In response to detecting the second input, the computer-generated environment is transitioned to a virtual reality environment while continuing to provide the communication session.

22. The method of claim 17, wherein the communication session is provided in response to receiving a request to initiate the communication session.

23. The method of claim 17, wherein generating the computer-generated environment comprises presenting a virtual representation of the remote user in the computer-generated environment.

24. The method of claim 17, wherein the computer-generated environment is generated in response to detecting an input to an object associated with an application presented using the electronic device.

Images