Virtual recommendation system

Abstract

A computer-implemented method for guiding a user to interact with an object is provided. The method includes: receiving an information cue of a scene including an object, the information cue relating to data retrieved by a sensor; prompting an identification object based on the information; extracting context information based on the information cue; determining at least one interaction indicator configured to indicate an interaction capable of being performed on the identified object, the at least one interaction indicator based at least in part on the contextual information; and guiding the user to perform an interaction with the object based on the at least one interaction indicator by controlling a mixed reality (MR) device.

Description

Background Art

[0001] Some existing e-commerce applications include features for viewing products and information about products in mixed reality (MR). Mixed reality (MR) includes a range of visualization technologies ranging from virtual reality (VR) to augmented reality (AR). For example, existing e-commerce applications include features for presenting virtual products using AR technology to present them to users in 3D. Such solutions allow users to digitally view products at a desired location. Similar solutions exist in VR. For example, a user can browse a virtual room or store with one or more products. In addition, some existing solutions use AR or VR to display additional product information.

[0002] In some existing AR or VR solutions, information about a product is displayed through buttons or links on 2D user interface elements. For example, the information can be displayed on a 2D panel shown near the product or on top of the product. In some examples, the information includes the product name, price information about the product, user reviews of the product, and promotional information. Summary of the invention

[0003] In general, the present disclosure relates to methods and systems for presenting virtual recommendations for an object. In some embodiments, the recommendations are presented in MR (including either AR or VR). Therefore, the present disclosure covers one or more sub-areas of extended reality (XR).

[0004] In a first aspect, a computer-implemented method for guiding a user to interact with an object is provided. The method includes: receiving an information prompt of a scene including the object, the information prompt being related to data retrieved by a sensor; identifying the object based on the information prompt; extracting context information based on the information prompt; determining at least one interaction indicator configured to indicate an interaction that can be performed on the identified object, the at least one interaction indicator being at least partially based on the context information; and guiding the user to perform an interaction on the object based on the at least one interaction indicator by controlling a MR device.

[0005] In a second aspect, an MR device is provided. The MR device includes a camera, a display, at least one processor, and a memory device, wherein the memory device stores instructions that, when executed by the at least one processor, cause the MR device to: obtain information cues of a scene; process the information cues to identify an object and extract context information; determine at least one interaction indicator configured to indicate an interaction that can be performed on the identified object, the interaction indicator being at least partially based on the context information; and guide a user to perform an interaction on the object based on the at least one interaction indicator by controlling the MR device.

[0006] In a third aspect, a server is provided that includes a processing unit configured to: process information cues received from an MR device to identify an object and extract context information; determine at least one interaction indicator configured to indicate an interaction that can be performed on the identified object, the interaction indicator being based at least in part on the context information; and transmit the at least one interaction indicator to the MR device to provide guidance to a user to perform an interaction on the object based on the at least one interaction indicator through control of the MR device.

[0007] In some examples, the processing is performed by respective processing devices of either or both of the MR device and the server (e.g., at least one processor of the MR device and a processing unit of the server). In these examples, the processing activity is not limited to a particular unit, as the computation can be performed at either or both of the server side and the client side. As an example, the MR device may perform the processing activity, but obtain the interaction indicator for a particular object through external input from the server or an external database separate from or integrated with the server. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An example environment for presenting virtual recommendations is shown.

[0009] Figure 2 An example server according to some implementations of the present disclosure is shown.

[0010] Figure 3 An example method for providing virtual recommendations is shown.

[0011] Figure 4 An example method for processing an image stream to identify objects is shown.

[0012] Figure 5 An example method for extracting contextual information is shown.

[0013] Figure 6 An example method for selecting recommendations for a subject is shown.

[0014] Figure 7 An example user computing device is shown in accordance with some implementations of the present disclosure.

[0015] Figure 8 An example method for presenting virtual recommendations in AR is shown.

[0016] Fig. 9 An example method for presenting recommendations in AR is shown.

[0017] Fig.10 An example method for presenting virtual recommendations in VR is shown.

[0018] Fig.11A An example user interface for an object recommendation application is shown.

[0019] Fig. 11B An example user interface for an object recommendation application is shown.

[0020] Fig.12 An example user interface for an object recommendation application is shown.

[0021] Fig.13 Example methods for determining and tracking lifecycle characteristics about an object are shown.

[0022] Fig.14 An example environment for presenting virtual recommendations is shown. DETAILED DESCRIPTION

[0023] Various embodiments will be described in detail with reference to the accompanying drawings, wherein similar reference numerals denote similar parts and components throughout the several views. Reference to various embodiments does not limit the scope of the appended claims. In addition, any examples set forth in this specification are not intended to be limiting, but merely set forth some of the many possible embodiments with respect to the appended claims.

[0024] This disclosure introduces many technical terms to fully describe the present invention and its various aspects, examples or embodiments. It should be noted that although specific terms are used herein for the sake of brevity, the terms in this disclosure may be interpreted in other ways by those skilled in the art.

[0025] In the examples of the present disclosure, the term "image stream" is defined. It should be noted that the image stream is an exemplary type of information prompt as image data. In other examples not explicitly stated herein, the information prompt may be considered to be other types of data, such as mesh data, auditory data, tactile data, environmental data, motion data, olfactory data, etc.

[0026] In examples of the present disclosure, the term "recommendation" is defined in the context of selecting a recommendation from a set of recommendations. It should be noted that a recommendation may indicate an interaction that may be performed on an identified object. To this end, a recommendation in the examples herein is an interaction indicator (i.e., an indicator configured to indicate in what manner a user of an MR device may interact with an object through control of the MR device).

[0027] An interaction may be any type of interaction that may be performed on an object, ranging from passive interactions to active interactions and anything in between. Passive interactions may include actions or engagements that may occur without direct or intentional input from a user, and are typically automatic or reactive in nature, requiring minimal user intervention. As examples, passive interactions may be notifications, updates, ambient displays of information, sensor-based triggers, proximity-based triggers, contextual adaptation, content recommendations, adaptive or predictive behavior, ambient audio, energy optimization, and the like.

[0028] Active interactions, on the other hand, may include actions or engagements initiated by a user to interact with the surrounding environment including the object, and typically include user input and control. As an example, active interactions may be physical or virtual object manipulation by means of physical or virtual touch, gesture control, voice commands, touch interactions, menu selections, drag-and-drop actions, camera interactions, physical control inputs, and the like.

[0029] The guidance according to the present disclosure reliably assists the user in performing technical tasks by means of guided human-computer interaction processing. The technical task to be performed depends on the object and context of the scene as determined based on the information prompt. For example, consider a scene in which the scene is a bathroom and the object is a specific electronic device. The information prompt can be received in the form of an image stream, and the electronic device can be identified from the image stream. In addition, the information prompt in the form of sensory input can be received from the humidity sensor, and context information can be extracted based on the information prompt to determine that the scene is a bathroom. This is because bathrooms are generally associated with higher humidity than other similar rooms. Based on the context information, an interaction indicator can be determined. In this particular example, the interaction indicator can provide an indication that the electronic device should be removed from the bathroom, because if the electronic device is kept in a humid condition corresponding to the bathroom, the electronic device may be at risk of damage. The user is then guided by the control of the MR device, for example, by a tactile input of a pair of tactile gloves, an auditory signal corresponding to a warning sound, an avatar that visually drags the electronic device out of the bathroom presented in AR, or any other similar indication to perform an interaction to remove the electronic device from the bathroom. Therefore, the user is guided to perform a technical task through human-computer interaction processing. Obviously, many different situations can be envisaged in which a user can be guided to perform an interaction with an object (as a technical task) based on an interaction indicator, which in turn is based on contextual information.

[0030] Guidance is typically performed by controlling the MR device. Control of the MR device may be implemented by the MR device itself, for example, the MR device is configured to execute one or more control commands corresponding to the interaction indicator. Alternatively or additionally, control of the MR device may be implemented by an external service, for example, the MR device can be controlled via one or more control commands. In this regard, the external service may be a server unit, such as a cloud-based computing resource, or other external units including other devices.

[0031] In the examples of the present disclosure, providing recommendations to a MR device, such as an AR, VR, or MR device, is described. It should be noted that providing recommendations using this method is an exemplary type of how a user can be guided to perform an interaction with an object based on at least one interaction indicator. Additionally or alternatively, the user can be guided to interact with the object in a variety of different ways, such as by receiving auditory or tactile feedback, etc.

[0032] In general, the present disclosure relates to methods and systems for guiding a user to interact with an object through control of a MR device. In some embodiments, the recommendation is presented in AR. In other embodiments, the recommendation is presented in VR.

[0033] Typically, guidance to the user is achieved by providing at least one interaction indicator (i.e., an indicator of an interaction that can be performed on the identified object). As will be apparent after the examples of the present disclosure, guidance can be provided in the form of recommendations for how the user can interact with the object through interaction.

[0034] Recommendations typically include information that conveys knowledge about a product to a retailer's customers. For example, product care information such as cleaning instructions may be presented to a customer using the product. Other information may also be presented along with the recommendation (e.g., materials, care instructions, disposal instructions, etc.).

[0035] In some embodiments, recommendations are selected by first identifying objects and contextual information in the scene. Contextual information may include features of the scene, features of objects in the scene, or features of objects related to or combined with features of the scene. For example, contextual information may determine that an object is in an external environment (features of the scene), that an object includes certain defects (features of the object), or that an object is too far from a supporting wall (features of the object relative to features of the scene). In one example, features of the scene include a room type, such as a bathroom. After identifying the objects and contextual information, recommendations are selected and presented in AR, VR, or MR. In some embodiments, scenes and objects are analyzed to determine a specific way to present the recommendation, such as presenting it in AR in a way that is unobtrusive to the user.

[0036] In some embodiments, an object is detected based on the gaze of a user wearing the MR device. The MR device, such as a pair of smart glasses, is equipped with gaze tracking technology including a scene camera and an eye camera. The eye camera is configured to monitor the movement and direction of the user's eyes. The scene camera is configured to capture the front of the user's field of view in the direction in which the user directs his gaze. The MR device may also be configured to apply a computer vision algorithm (optionally using machine learning techniques) to analyze and determine the visual input captured by the eye camera. The MR device may also be configured to map the input obtained by the scene camera with the input obtained by the eye camera to identify the object, and accordingly determine the interaction indicator based on the object the user is looking at and the associated contextual information.

[0037] In many embodiments, machine learning is used to identify features in an information prompt (e.g., an image stream), including algorithms for identifying objects and contextual information. In addition, in some embodiments, machine learning is used to determine where to present at least one interaction indicator, such as a recommendation. In some embodiments, a machine vision algorithm identifies an object in an image stream. In some embodiments, the machine vision algorithm also identifies features of an object. In some embodiments, features are also used to determine where to present a recommendation. In some embodiments, contextual information is identified using a machine learning algorithm that predicts contextual information based on features identified in an information prompt (e.g., an image stream). For example, the intended use of an object can be predicted based on the environment of the detected object, such as predicting that a user will wash an object based on detecting that the object is placed near a sink, and the recommendation will include washing instructions given to the user in AR.

[0038] In some embodiments, recommendations are presented to provide the most useful, relevant, or important information to the user based on the contextual information. For example, when the contextual information identifies a potential safety issue, safety information may be presented, but if no safety issue is detected, subject care recommendations are provided. In some examples, the recommendations provided to the user are predefined based on a set of rules. In other embodiments, a machine learning algorithm may be used to score the recommendations based on the identified contextual information as well as other input data.

[0039] Reference will now be made to the accompanying drawings following this paragraph. The specific exemplary embodiments described with reference to the accompanying drawings relate to information prompts in the form of image streams. In addition, a set of recommendations is defined, wherein a specific recommendation is selected from this set of recommendations. The selected recommendation is then sent to the AR device so that the recommendation can be presented in AR. As discussed above, the skilled person will understand other types of information prompts not explicitly described herein. In addition, the skilled person will understand that at least one interaction indicator configured to be executable on the identified object is a type of recommendation. Therefore, it is not necessary to determine a set of recommendations, and it is not necessary to select a recommendation from it. Alternatively, other conceivable examples include determining at least one interaction indicator, and guiding the user to interact with the object through interaction based on the determined at least one interaction indicator.

[0040] Figure 1 An example environment 100 for presenting virtual recommendations is shown. Environment 100 includes a user computing device 102 connected to a server 104 via a network 122. In some implementations, the user computing device includes a camera 108 that captures a scene 106. In some implementations, scene 106 can be captured by multiple user computing devices including corresponding cameras, and scene 106 can be constructed by combining inputs from corresponding cameras of multiple user computing devices. In other implementations, scene 106 is a virtual scene presented on user computing device 102. User computing device 102 includes an object recommendation application 110 that presents recommendations 112 in AR, MR, or VR. Server 104 includes a recommendation engine 114 and an object recommendation data store 116. Scene 106 includes object 118.

[0041] The user computing device 102 operates to present virtual recommendations for an object. In some implementations, the user computing device 102 is an AR device, such as a smartphone, a tablet, smart glasses, etc. In some implementations, the user computing device 102 is a VR device, such as a VR headset, a smartphone, a tablet, etc. In some implementations, the user device includes a camera 108 and a display 109.

[0042] In an AR implementation, a camera 108 captures an image stream of a scene 106. The image stream is received by an object recommendation application 110, which communicates with a server 104 to operate a recommendation engine 114. The recommendation engine 114 processes the image stream to determine recommendations for objects. In a VR implementation, a camera 108 is not required. A display 109 presents virtual recommendations in AR, VR, or MR.

[0043] As discussed above, an image stream can be a type of information received as an information cue. In general, the term information cue refers to a visual or perceptual element present in a scene that can provide information (or a cue) to a device. The information cue can be derived from sensory input received by a device such as an MR device. The information cue may include one of image data (i.e., an image stream), mesh data, auditory data, tactile data, environmental data, motion data, olfactory data, or any combination thereof. The mesh data in this context may include vertices, edges, faces, or normals, etc., of objects or surfaces used to represent a scene. Any information cue that can assist in the recognition of an object may be included.

[0044] The user computing device 102 operates the object recommendation application 110. The object recommendation application displays the recommendation 112. In some embodiments, the recommendation is presented as part of the 3D object. In some embodiments, the 3D recommendation is presented in a realistic manner. For example, the 3D recommendation can be presented on a 3D virtual label attached to the object. In some embodiments, the recommendation overlays a portion of the object. In some embodiments, the recommendation includes information that would be presented on the packaging of the product. Figure 7 Examples of user equipment are shown and described.

[0045] In some embodiments, the recommendations 112 for the object may include recommendations related to the material of the object, object size, object care instructions, object disposal information, object sustainability information, assembly information, installation error information, consumption / replacement information, object intended use information, or any combination thereof.

[0046] The server 104 operates to process the image stream of the captured scene 106 to identify relevant recommendations for the object 118. In some embodiments, the server 104 is part of a retailer system and / or an e-commerce system. Although only one server is shown, some embodiments include multiple servers. In these embodiments, each of the servers can be the same or similar and can provide similar functionality (e.g., to provide greater capacity and redundancy, or to provide services from multiple geographic locations). Alternatively, in these embodiments, some of the multiple servers can perform dedicated functions to provide dedicated services. Various combinations thereof are also possible. Reference Figure 2 An example of server 104 is shown and described.

[0047] The server 104 includes a recommendation engine 114. The recommendation engine 114 selects recommendations 112 for an object 118. In some implementations, the recommendation engine 114 processes the image stream to identify objects and extract contextual information. The objects and contextual information are used to select recommendations that are relevant to the user. Example methods for identifying objects, extracting contextual information, and selecting recommendations are described herein.

[0048] The server 104 includes or interfaces with an object recommendation data store 116. The object recommendation data store 116 includes one or more recommendations for a plurality of objects. In some embodiments, the object recommendation data store 116 is the same data store that presents object information on a retail website and / or stores a digital brochure of a plurality of objects provided by a retailer.

[0049] Environment 100 may include scene 106, which may be a virtual reality scene or an augmented reality scene. An augmented reality scene may include any physical space. For example, an augmented reality scene may include a room in a house, an office, a store, a warehouse, a backyard, an event space, an event / conference venue, a staged model home / apartment, etc. In other examples, scene 106 is a virtual reality scene, such as a virtual reality store or a virtual reality home. In some embodiments, the scene is presented with a furniture planner.

[0050] Scene 106 includes object 118. In the example shown, the object is a cutting board with a virtual recommendation 112 for hand-washing cutting boards. In some embodiments, object 118 can be any object. In other embodiments, object 118 is a product sold by a specific retailer. Examples of object 118 include furniture, electronic devices, musical instruments, food, plants, medicine, household items (e.g., tableware, napkins, candles, pots), etc. Object 118 can be part of a larger object. For example, object 118 can be a shelf on a cabinet or a cushion on a sofa.

[0051] The network 122 connects the server 104 to a plurality of computing devices including the user computing device 102. In some examples, the network 122 is a public network, such as the Internet. In an example implementation, the network 122 may connect to the computing devices through a Wi-Fi network or a cellular network.

[0052] Figure 2 An example server 104 is shown. In some embodiments, the server includes a processor 142, a memory 144, and a network interface 146. In some embodiments, the server 104 interfaces with an e-commerce system or platform. In other embodiments, the server is integrated with the e-commerce system. The memory stores instructions for executing the recommendation engine 114. The recommendation engine 114 includes an object identifier 152, a recommendation selector 154, and a context identifier 156. In some embodiments, the recommendation engine 114 interfaces with the object recommendation data store 116, the customer data store 140, and the 3D model data store 164.

[0053] The processor 142 includes one or more central processing units (CPUs). In other embodiments, the processor 142 includes one or more digital signal processors, field programmable gate arrays, or other electronic circuits. In some embodiments, the processor includes one or more processors (e.g., virtual or physical processors) that execute instructions for executing an algorithm to achieve a desired result. In addition, in some embodiments, additional input / output devices are operably connected to the processor 142.

[0054] The memory 144 is operably connected to the processor 142. The memory 144 typically includes at least some form of computer-readable media. Computer-readable media may include computer-readable storage media and computer-readable communication media.

[0055] Computer-readable storage media include volatile and nonvolatile, removable and non-removable media implemented in any device configured to store other data. Computer-readable storage media include, but are not limited to, random access memory, read-only memory, flash memory and other memory technologies, compact disc read-only memory, Optical discs, digital versatile discs or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store the desired information and that may be accessed by server 104. In some implementations, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0056] Computer-readable communication media typically embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in such a way as to encode information in the signal. As an example, computer-readable communication media include wired media such as a wired network or a directional wired connection, and wireless media such as acoustic, radio frequency, infrared, and other wireless media. Combinations of any of the above are also included within the scope of computer-readable media.

[0057] A number of program modules may be stored in memory 144 or in a secondary storage device, including an operating system, one or more application programs and other program modules, and program data. Memory 144 stores instructions for the recommendation engine.

[0058] The recommendation engine 114 includes an object identifier 152, an object recommendation selector 154, and a context identifier 156. In some implementations, the recommendation engine 114 interfaces with the object recommendation data store 116, the customer data store 140, and the 3D model data store 164.

[0059] The network interface 146 operates to enable the server 104 to communicate with the server 104 via one or more networks (eg, Figure 1 The network 122 shown in FIG. 1 communicates with one or more computing devices. For example, the server 104 is configured to communicate with the user computing device 102 to perform some of the methods disclosed herein. The network interface may include a wired and / or wireless interface.

[0060] In some embodiments, the recommendation engine 114 interfaces with one or more data stores. In some embodiments, the data store is associated with an e-commerce system or platform that stores product information for one or more retailers. In the example shown, the data store interface operates to send and receive data from the object recommendation data store and the customer data store.

[0061] In some embodiments, the object recommendation data store 116 is a centralized retail data store that includes information about products sold by retailers, including product details, product materials, product care instructions, care instructions, safety information, compliance information, sustainability information, recycling information, assembly information, measurements, object weights, user reviews, images of products, pricing information, user manuals, etc. In some embodiments, the object recommendation data store 116 stores recommendations and object information in multiple different languages.

[0062] In typical implementations, the recommendation engine 114 need not interface with the customer data store 140. However, in some implementations, the recommendation engine 114 interfaces with the customer data store 140 to retrieve customer information. For example, the customer data store 140 may track products sold to customers and use these products to assist in identifying objects or for identifying contextual information.

[0063] In some embodiments, the 3D model data store stores 3D models for a plurality of objects. In some embodiments, the 3D models are annotated based on features to allow the object identifier 152 to quickly filter the models based on a set of features identified in the input image stream.

[0064] Figure 3 An example method 208 for providing virtual recommendations is shown. In some implementations, the method 208 operates as part of the recommendation engine 114 at the server 104. The method 208 includes operations 210, 212, 214, 216, and 218.

[0065] In an AR implementation, operation 210 receives an image stream capturing a scene having at least one object. In some implementations, the image stream is continuously processed to update as the user navigates through the scene. In a VR implementation, data defining the scene is received instead of an image stream.

[0066] Operation 212 processes the image stream to identify at least one object. In some embodiments, a machine vision algorithm is used to identify the object. In some embodiments, the image stream is processed to identify a 3D shape and a visual search algorithm is used to match the 3D shape to a 3D model of the object. Figure 4 An example method for identifying objects in an image stream is shown and described.

[0067] Operation 214 retrieves a set of recommendations for each of the at least one object. In some embodiments, the set of recommendations is retrieved from a data store storing object information. In some embodiments, the set of recommendations is predefined. In other embodiments, a machine learning model extracts the recommendations from data about the objects. In an alternative embodiment, the set of recommendations is parsed for a retailer's website. The set of recommendations may include different languages ​​and / or 3D symbols or pictures representing the recommendations. For example, recommendations about how to use an object may include an image of an object with defects caused by improper use.

[0068] Operation 216 processes the image stream to extract contextual information. In some embodiments, the contextual information is related to a scene, an object, or a combination of a scene and an object. In some embodiments, the contextual information is also based on additional data. For example, location data (e.g., as determined using GPS data at a GPS receiver) is used to determine the location. The location may determine that the user is at a retailer's showroom, and the contextual information includes known features of the showroom. The contextual information may also be based on the condition of the object. For example, the contextual information may include the current life cycle period of the object. In some embodiments, the life cycle period is defined as the different stages that an object goes through from the creation of the object to the destruction of the object. In some embodiments, each period is defined by the type of recommendations associated with the object at different stages in the life cycle of the object. For example, the life cycle period of an object that is at or near the end of its life will include instructions for the disposal or recycling of the object. The contextual data may also include user data, such as the date the item was purchased, the preferred language, a floor plan annotated for the user, a VR layout of the store with various objects / products, and the like. Reference Figure 5 Example methods for extracting contextual information are shown and described.

[0069] Additional examples of contextual information that may be extracted include: setting, geographic location, life cycle period of an object, environment type; object defects, object use or misuse, object safety issues; or any combination thereof.

[0070] Operation 218 selects a recommendation based at least in part on the extracted contextual information. For example, if the contextual information indicates that the user is about to wash the object, operation 218 will select a recommendation with washing instructions. Similarly, if the contextual information indicates that the user is installing the object, operation 218 may select installation instructions. In some embodiments, a set of rules defines which recommendation is selected. For example, a rule may define a default recommendation and one or more recommendations that are displayed in certain circumstances. In some embodiments, when a potential safety issue is detected, the rule prioritizes safety recommendations. For example, when a bookshelf that needs to be anchored to a wall is identified as not being located next to a wall, a recommendation that includes a safety warning is selected.

[0071] In some embodiments, a machine learning model is used to select recommendations. In one embodiment, a machine learning model can be trained based on customer reviews and object return data to determine defects or problems with an object and select recommendations that can prevent common defects. In some embodiments, images of objects with deformed shapes are saved, including images of returned products or images of defective products submitted by customers.

[0072] The machine learning model can be implemented to identify anomalies of an object. Anomalies are indications that an object is different from its expected appearance and / or function. In order to identify anomalies, the machine learning model can be trained based on the object data of multiple objects. The machine learning model can be configured to receive anomaly identification requests for an object of interest, for example, in response to detecting an object of interest. The machine learning model trained based on the object data can be configured to process the anomaly identification request to determine whether the object of interest is associated with the anomaly. In response to having processed the anomaly identification request, feedback on the results of the anomaly identification request can be provided to the user, indicating whether the object of interest is associated with the anomaly. To this end, the machine learning model is configured to predict anomalies based on previously performed recognition. The machine learning model can also analyze the results of the anomaly identification request to improve further anomaly classification. By way of example only, the machine learning model can use supervised or unsupervised learning algorithms known in the art, including but not limited to neural networks, binary, multi-class or multi-label classification, clustering algorithms, regression algorithms, support vector machines, kernel estimation, decision trees, etc.

[0073] Operation 220 sends each recommendation to the user device to display the recommendation attached to the corresponding object. In some implementations, the recommendation may be sent along with instructions on where to place the recommendation and the type or style of 3D element used to present the recommendation.

[0074] In many implementations, method 208 is repeated as the user browses the scene providing a nearly continuously updated stream of images.

[0075] Figure 4An example method 212 for processing an image stream to identify an object is shown. The method 212 is a reference to Figure 3 An example of operation 212 is shown and described. Method 212 includes operations 244, 246, and 248.

[0076] Operation 244 identifies visible surfaces of objects in the image stream. In a typical implementation, a machine vision algorithm is used to identify visible surfaces in the image stream.

[0077] Operation 246 compares the identified surfaces to the 3D model of the object. The identified surfaces are compared to the 3D model using a machine vision algorithm. In some embodiments, the 3D model is associated with a product sold by a particular retailer. In some embodiments, the 3D model is annotated and indexed to allow the machine vision algorithm to quickly compare visible surfaces to related 3D models. For example, the 3D model can be indexed based on room type and the 3D model can be annotated based on typical positioning of the object. In this example, the machine vision algorithm can identify the room type and filter the model based on the identified room type, and then compare the surfaces indicated as potentially visible.

[0078] Operation 248 identifies one or more objects based on the comparison with the 3D model. In some implementations, operation 248 predicts the likelihood that the visible surface is a particular object, and if the likelihood is above a set threshold, operation 248 will identify the object.

[0079] In some embodiments, specific surfaces of an object are identified. For example, recommendations can be provided for specific surfaces or materials of an object. In some embodiments, identifying specific surfaces is accomplished by first identifying the object and retrieving a 3D model of the object. The 3D model of the object includes annotated surfaces, and it is used to map the annotated surfaces with the identified object. For example, the legs of a table may be annotated with material information, such as a different type of metal than the surface of the table. Identifying specific surfaces or types allows the methods and systems to provide surface or portion specific recommendations. Advantages of identifying an object before identifying specific surfaces include increased accuracy and efficiency in identifying specific surfaces.

[0080] Figure 5 An example method 214 for extracting context information is shown. The method 214 is for referring to Figure 3 An example method of operation 214 is shown and described. Method 214 includes operations 260, 262, and 264.

[0081] Operation 260 retrieves object data for each identified object. In a typical embodiment, the object data is retrieved from a centralized data store that includes object information for a large collection of objects. In some embodiments, the collection of objects includes objects sold by a particular retailer. In some embodiments, the object data also includes annotated images of objects with defects. The annotations may describe what problem caused the defect and recommendations for avoiding the defect. In a typical embodiment, the object information will include a collection of recommendations for each of the objects.

[0082] Operation 262 processes the image stream using one or more machine vision models. The machine vision model may include additional inputs to improve the accuracy of the predicted contextual information. For example, objects purchased by the user, location data, environmental data (e.g., humidity, temperature as determined by sensors connected to the user's device) may be used as inputs for extracting contextual information.

[0083] Operation 264 extracts contextual information based on the output of one or more machine vision models, customer data, and / or product data. In some examples, the contextual information is also based on tracking the user's location in the environment. For example, a user can draw a map of their house and annotate different rooms, and based on the image stream and the annotated map, operation 264 determines which room the user is in. In addition, the contextual information may also include information extracted from user comments and identified in the image stream. For example, if a user comment indicates that a product is deformed due to being placed outside, this information can be extracted and compared to the environment identified in the image stream to provide care recommendations for avoiding defects. Contextual information may also be based on system settings, such as language or accessibility settings.

[0084] Figure 6 An example method 216 for selecting a recommendation for an object is shown. In some embodiments, the method 216 is performed with reference to Figure 3 A portion of operation 216 is shown and described. In a typical implementation, operation 282 is performed at a server having a recommendation engine. However, in some implementations, some or all of the operations are performed locally at a user computing device. Method 216 includes operations 282, 284, 286, and 288.

[0085] Operation 282 retrieves a set of recommendations based on the identified object. In some embodiments, the database stores recommendations for each of the available objects. In other embodiments, recommendations for products corresponding to the identified object can be collected from retailer websites.

[0086] Operation 284 selects a recommendation from the set of recommendations based at least in part on the contextual information. The contextual information is used to determine which recommendations may be relevant to the user at a particular time. In some embodiments, a trained machine learning model is used to select the recommendation. The machine learning model can be trained based on user interactions with the application and customer reviews of the object. In other embodiments, a set of rules or policies for selecting recommendations are defined. In some embodiments, each recommendation is scored based at least in part on the contextual information, and the highest scoring recommendation is selected. The machine learning model can utilize any suitable algorithm commonly used in the art, such as the algorithms discussed above.

[0087] Operation 286 selects a recommendation element type for presenting the selected recommendation. In some embodiments, the element type is a 3D element, which corresponds to the way the user desires the information to be physically presented on the object. In other embodiments, the element is selected to minimize occlusion of the object or scene. For example, the recommendation may be printed on a flat surface of the object (e.g., as shown in FIG. Fig.11A In other embodiments, recommendations are presented near objects to avoid occluding the objects (e.g., as shown in Fig. 11B ).

[0088] Operation 288 generates a recommendation using the selected recommendation element and the selected recommendation. In some embodiments, an AR engine on the user computing device generates graphics overlaying the real-time image stream to present the recommendation in AR. In some embodiments, a VR engine places a virtual recommendation in a VR scene.

[0089] In some embodiments, identifying a recommendation includes analyzing the environment surrounding the object. For example, an object is identified and the object is approved for indoor use but not approved for use in a bathroom (or other room with high humidity), and it is found from contextual information (e.g., from other objects identified in the scene from the image stream) that the object is in a bathroom environment, so the user is then recommended to move the product to another environment (or other products approved for bathroom use are recommended). Another example includes identifying an object that needs to be attached to a wall for safety, but identifying that the object is not placed close enough to any wall, so the user is recommended to anchor the object to the wall. Another example includes identifying an object that is not approved for children in an environment with other objects approved as children's products, and notifying the customer that the product is not approved for children.

[0090] In some implementations, the recommendations are translated into a specific language based on user account settings, system settings of the user's computing device, and / or GPS data. In some implementations, the recommendations are presented as 3D symbols.

[0091] Figure 7An example user computing device 102 is shown. The user computing device 102 includes a processor 302, a camera 108, a display 306, a network interface 308, and a memory 310. The memory stores instructions for the object recommendation application 110 and an AR / VR engine 312. The AR / VR engine 312 includes a recommendation placer 314 and a recommendation type selector 316.

[0092] The user computing device 102 includes a processor 302, a network interface 308, and a memory 310. Examples of processors, memories, and network interfaces are described herein. For example, refer to Figure 2 Processor 142 , memory 144 , and network interface 146 are shown and described.

[0093] In some implementations, the user computing device 102 includes a camera 304. The camera 108 is used to capture images of a scene. The camera 108 can be any type of camera commonly used on mobile computing devices (including augmented reality devices). In the VR example, the camera 108 is not required.

[0094] Other sensors may also be used to capture 3D details of the physical environment. For example, LIDAR sensors, ultrasonic distance sensors, or using sounds generated by a speaker and analyzing the echoes received at a microphone. In other embodiments, images from two or more cameras are used to calculate 3D features in the physical environment.

[0095] Display 109 can be any electronic display capable of presenting virtual recommendations. In some examples, the display is a screen, such as a touch screen on a mobile device, a television, a monitor, a projector, a hologram, etc. In some examples, the display is specifically designed for use in augmented reality and / or virtual reality.

[0096] A number of program modules may be stored in the memory 310 or in a secondary storage device, including an operating system, one or more application programs and other program modules, and program data. In the example shown, the memory 310 stores instructions for the object recommendation application 110 .

[0097] The object recommendation application 110 operates to present recommendations to the user. In some implementations, the object recommendation application 110 is an AR application. In other implementations, the object recommendation application 110 is a VR application.

[0098] The AR / VR engine 312 operates logic for presenting recommendations in an AR or VR manner. The AR / VR engine 312 includes a recommendation placer 314 and a recommendation type selector 316.

[0099] The recommendation placer 314 determines a location for placing the virtual recommendation. In some implementations, the recommendation is placed in a location that is easily viewed by the user while minimizing occlusion of the object and the surrounding scene. Fig. 9 An example method performed by a recommendation placer 314 is shown and described. In a typical implementation, the recommendation placer 314 determines where and how to display the recommendation. In some implementations, the location is manually defined based on the surfaces that are most likely to be used or displayed. For example, a 3D model of an object may include annotations that define the surfaces that are most often used or displayed.

[0100] In some implementations, the recommendation placer 314 determines a specific surface of the object to place a recommendation associated with the specific surface of the object. For example, the specific surface of the object may include a recommendation with specific care instructions (e.g., based on material, design, etc.).

[0101] In some embodiments, multiple recommendations are presented on an object to be attached to different surfaces. For example, multiple surfaces of different materials are identified, and a recommendation is selected for each of the different surfaces. The recommendation is placed to be attached to the corresponding surface.

[0102] The recommendation type selector 316 selects an element for presenting the recommendation. The element can be a 2D or 3D element. In some embodiments, a 3D element is a replica of a real-world element used to attach information to an object. For example, a sofa cushion may include a label, and the 3D element will match the label with the information. In other embodiments, the element is selected to clearly present the recommendation while minimizing obstruction to the user's view.

[0103] Figure 8 An example method 320 for presenting virtual recommendations in AR is shown. In some embodiments, the method 320 is performed on a user computing device 102 (e.g., as shown in FIG. Figure 1 and Figure 7 Method 320 includes operations 322, 324, and 326.

[0104] Operation 322 captures an image stream of the scene. In a typical implementation, the user device includes a camera that captures an image stream that is processed and updated as the user views the scene.

[0105] Operation 324 provides the image stream to a recommendation engine to process the image stream to identify objects and extract context information from the image stream, retrieve a set of recommendations for the objects, and select a recommendation from the set of recommendations based on the identified objects and the context information. Figure 3 An example method of operation 324 performed at a recommendation engine is shown and described.

[0106] Operation 326 receives the recommendation and presents the recommendation attached to the object in AR. Fig. 9 An example method for presenting recommendations in AR is shown and described.

[0107] Fig. 9 An example method 326 for presenting recommendations in AR is shown. In some embodiments, the method 326 is a reference to Figure 8 A portion of operation 326 is shown and described. In some implementations, method 326 includes operations 342, 344, 346, 348, and 350.

[0108] Operation 342 identifies visible surfaces on the object. In some implementations, the image stream is processed to determine if there are any objects that occlude any portion of the object.

[0109] Operation 344 determines the pose of the object. In some implementations, the visible surface of the object is analyzed to determine the current position and orientation of the object.

[0110] In some embodiments, method 326 includes an operation 346 of determining the distance to the object. In some embodiments, a machine vision algorithm calculates the distance based on the received image stream. In a typical embodiment, operation 346 is not required. In some embodiments, the type of element used to present the recommendation is updated based on the distance of the user from the object.

[0111] In some embodiments, method 326 includes an operation 348 of analyzing lighting conditions of the scene. In typical embodiments, operation 346 is not required. The lighting conditions are analyzed so that the recommendations are placed in locations with neutral lighting.

[0112] Operation 350 selects a location for presenting information based on the visible surface of the identified object, the pose of the object, the distance to the object, and / or the lighting conditions of the scene. In some embodiments, each object included in the object data store includes a set of predefined points for presenting virtual recommendations. In some embodiments, each predefined point is scored based on the visible surface, the pose of the object, the distance from the object, and / or the lighting of the scene. For example, a point visible to the user's line of sight on the front surface is selected to virtually associate a 3D element with the recommendation. In some embodiments, a set of rules defines where the recommendation is placed. For example, a rule may define a visible point selected on the front surface or closest to the front surface. In some embodiments, the distance from the object is used to determine the size of the 3D element at which the recommendation is presented. This allows the recommendation to be adjusted to provide useful information to the user without obstructing the user's line of sight to the scene.

[0113] Fig.10An example method 370 for presenting virtual recommendations in VR is shown. In some implementations, the method 370 is performed on a user computing device 102 (e.g., as described in reference to Figure 1 and Figure 7 Method 370 includes operations 372, 374, and 376.

[0114] Operation 372 presents a virtual scene, which includes objects with object IDs. In many embodiments, multiple objects each having an object ID are presented. In some embodiments, the virtual scene is a customized home layout. In some embodiments, the virtual scene is a virtual showroom.

[0115] Operation 374 sends the object ID to a recommendation engine, wherein the recommendation engine determines contextual information, retrieves a set of recommendations for the object via the object ID, and selects a recommendation from the set of recommendations based at least in part on the contextual information. In some embodiments, the VR scene is mapped with different contextual information depending on the user's current line of sight, the object ID, or a stored variable. In some embodiments, the contextual information includes an indication that the user is viewing the scene in VR. For example, cleaning instructions may not be relevant to a user viewing the virtual object. Figure 3 An example method performed at a recommendation engine is shown and described.

[0116] Operation 376 presents the recommendation attached to the object in the virtual reality scene. In some embodiments, the recommendation is attached to the object as it would be in the real world scene. For example, a virtual label may be attached to the object or virtual packaging. In some embodiments, the recommendation is presented in 3D near the object to avoid occluding the object.

[0117] In some VR implementations, the VR scene is connected to the user's account. For example, a user may have one or more VR scenes (e.g., a virtual home, a virtual office, etc.) connected to an account. When a user makes a purchase from a retailer connected to an account, information about the object purchased by the user is sent to the account and added to one or more scenes in the VR scene. In some implementations, the user can add the purchased object to a customized scene. In some implementations, the user can interact with or use the object while browsing the virtual scene. For example, if a user has an account with a virtual kitchen and they have purchased a cutting board, a virtual object in the form of a virtual cutting board can be placed in the kitchen. The user can pick up the virtual cutting board while browsing the virtual scene, and a virtual recommendation is displayed. In some implementations, the recommendation is based on contextual information such as when the object was purchased. For example, if the cutting board was purchased a year ago, the recommendation can provide care instructions related to the year (e.g., a recommendation to oil the cutting board). In another example, if the user places a virtual product in a virtual scene that is not suitable for the product, the recommendation for moving the object is selected and presented to the user in VR. Many other examples of contextual information can be extracted from VR scenes.

[0118] Fig.11A An example user interface is shown for an object recommendation application 110. In the example shown, recommendations 112 overlay objects in an unobtrusive manner.

[0119] Fig. 11B An example user interface of an object recommendation application 110 is shown. In the example shown, recommendations 112 are presented near an object with a line connecting the recommendation 112 to the object. In this way, the recommendation is unobtrusive to the object.

[0120] Fig.12 An example user interface of an object recommendation application 110 is shown. In this example, the recommendation provides instructions on how to use the object. For example, the recommendation includes settings for preparing different types of food. In a related embodiment, the context information can be used to determine the type of food, and the recommendation will provide specific instructions for the identified type of food.

[0121] Fig.13 An example method 380 for determining and tracking lifecycle characteristics about an object is shown. The method 380 includes operations 382, ​​384, 386, 388, 390, 392, 394, and 396. The method 380 shows different recommendations presented based on a particular lifecycle period.

[0122] Operation 382 determines that the object is displayed for sale, and in response, operation 384 selects and displays a recommendation for the object. For example, information of interest to a potential purchaser of the object is displayed as part of the recommendation. For example, the purchaser may be interested in learning about the materials used in the object or sustainability information to help decide which object to purchase. In some embodiments, the object is displayed for sale in a virtual environment.

[0123] Operation 386 determines that the object is being installed, and in response, operation 388 selects and displays object installation and / or object safety information. In some embodiments, the recommendations are updated as the user completes the installation steps. In some embodiments, the recommendations include warnings to the user when installation errors occur.

[0124] Operation 390 determines that the object is in use, and in response, operation 392 selects and displays object care information. For example, when a cutting board is in use, a recommendation is provided to the user to wash both sides of the cutting board. Many other examples are disclosed herein.

[0125] Operation 394 determines that the object is at end of life, and in response, operation 396 selects and displays object disposal / recycling information. In some implementations, operation 396 queries local recycling and disposal rules and presents disposal recommendations based on the local rules.

[0126] In some embodiments, the guidance provides a series of recommendations to the user at different points in the life cycle. For example, a collection of recommendations can be given to the user as tasks to review at different life cycle stages. In some embodiments, upon unboxing the object, the recommendations are automatically displayed on the user's device. Examples of life cycle periods include: when the object is displayed for sale, when the object is being installed; when the object is in use, and when the object is at the end of its life.

[0127] Fig.14 An example environment 400 for presenting virtual recommendations is shown. The environment 400 includes a user computing device 102 and a server 104 connected via a network 122. The server includes a recommendation engine 114 and an object recommendation data store 116. The environment 400 is similar to Figure 1 The environment 100 shown in FIG. Fig.14 In the example of , the environment is a showroom. The selected recommendations can also be based on the environment being a showroom. For example, recommendations for specific materials are presented to a user viewing a specific object. In some embodiments, the environment 400 is a virtual showroom. In addition Figure 1 and Fig.14 In addition to the examples shown in , the systems and methods disclosed herein may be used in a variety of different locations, including a home, an office, a store, or another public or private space where an object of interest is located.

[0128] In some embodiments, a computer-implemented method for guiding a user to interact with a product is disclosed. The method includes: (1) identifying an object, matching the object to a model stored in a database, and providing information about the object. The information about the object may include one or more of care instructions, an intended use of the object, or an intended age range for use of the object. In some embodiments, the computer-implemented method includes identifying instructions on how to handle or care for the product by comparing a current shape or condition of the product with an original shape or condition of the product (e.g., from a database). In some embodiments, the instructions recommend an appropriate action based on comparing the current shape or condition of the product with the original shape or condition of the product.

[0129] Additional example implementations of some embodiments of the present disclosure include: (1) identifying the surface of an object that is associated with specific care instructions (e.g., a cutting board needs to be oiled every 6 months); (2) identifying that a specific surface is substantially different from the original surface (e.g., a curved cutting board - compare the scan of the object to the original 3D file) and recommend an appropriate action (e.g., always wash both sides of the board); (3) identifying different parts of an object and connecting a recommendation to a specific part (e.g., by highlighting that part, covering that part, etc.); (4) identifying an object and the object is approved for indoor use but not approved for the bathroom category, and discovering from context (from other products in the scanned scene) that it is in a bathroom environment, then recommending that the user not use the object in that environment (or recommending another product that is approved for bathroom use), (5) identifying consumables and instructions on when / how to change, identifying products that need to be attached to a wall for safety but identifying that the object does not have sufficient (6) identifying a product that is not approved for children in an environment with other products that are approved as children's products and notifying the customer that the product is not approved for children; (7) extracting contextual information, including that the room type is a kitchen with an identified object that is not approved for food, and selecting a recommendation to provide a warning to the user; (8) providing a recommendation to store food on shelves in the refrigerator (e.g., on shelves designed to store specific types of food); (9) identifying loose cords or old-fashioned blinds with cords near a child's bed and providing a warning; (10) identifying a mattress directly on the floor and providing a warning that the arrangement does not provide proper ventilation (e.g., legs are required); (11) presenting instructions for oil designed to maintain the object or recommendations for replacement parts; and / or (12) providing product recommendations when shopping (e.g., identifying areas of wear, etc.), including the condition of used products.

[0130] Further alternative aspects of the disclosure are described in the following numbered clauses.

[0131] Item 1: A method for guiding a user to interact with an object, the method comprising: receiving an image stream of a scene including an object from a camera of an augmented reality (AR) device; identifying the object in the image stream; retrieving a set of recommendations for the object from an object recommendation data store; extracting context information from the image stream; selecting a recommendation from the set of recommendations based at least in part on the context information; and sending the recommendation to the AR device for presentation in AR on a display of the AR device.

[0132] Clause 2: A method according to clause 1, wherein the context information may include one or more of the following: (1) settings; (2) geographic location; (3) object life cycle period; (4) environment type; (5) object defects; (6) object use or misuse; (7) object safety issues; or (8) any combination of the foregoing (1), (2), (3), (4), (5), (6) and (7).

[0133] Clause 3: A method according to clause 1, wherein the context information includes an object life cycle period, wherein the object life cycle period includes one of the following: (1) the object is displayed for sale; (2) the object is during installation; (3) the object is in use; and (4) the object is at the end of its life.

[0134] Clause 4: The method of clause 1, wherein the AR device determines a surface of the object to present the recommendation in AR, and wherein the recommendation is presented attached to the determined surface.

[0135] Clause 5: The method of clause 4, wherein the surface is determined by: determining a pose of an object and an unobstructed surface of the object based on an image stream of the scene; and determining the surface based on the determined pose of the object and the determined unobstructed surface of the object.

[0136] Clause 6: The method of clause 5, wherein the surface is further determined by: determining a distance to an object and a lighting condition of the scene, wherein determining the surface is further based on the distance to the object and the lighting condition of the scene.

[0137] Clause 7: The method of clause 1, wherein selecting a recommendation comprises ranking the set of recommendations, wherein the ranking is based at least in part on the contextual information, and the selected recommendation is the highest ranked recommendation.

[0138] Clause 8: The method of clause 7, wherein ranking the set of recommendations comprises processing the set of recommendations using a machine learning model to score the set of recommendations based at least in part on contextual information.

[0139] Clause 9: The method of clause 1, wherein when the extracted contextual information includes a recognized room type in the scene other than a recommended room type for the object, the selected recommendation includes an object warning.

[0140] Clause 10: The method of clause 1, further comprising: identifying at least two surfaces of the object; selecting a surface recommendation for each of the at least two surfaces; and sending the surface recommendation to an AR device, wherein each surface recommendation is presented in AR and attached to a corresponding surface of the at least two surfaces.

[0141] Clause 11: The method of clause 10, wherein each of the at least two surfaces is made of a different material, and the recommendation selected for each of the at least two surfaces includes care information about each of the different materials.

[0142] Clause 12: The method of clause 1, further comprising: identifying installation errors based at least on the image stream, the object, and the context information.

[0143] Clause 13: The method according to clause 1 also includes: identifying a surface of an object; comparing the surface on the object with a model stored in a database to identify defects in the surface; and sending a message to an AR device to present a recommendation related to the defect attached to the surface in AR.

[0144] Item 14: A method according to item 1, wherein the recommendation may include: (1) object material; (2) object dimensions; (3) object care instructions; (4) disposal information; (5) sustainability information; (6) assembly information; (7) installation error information; (8) consumption / replacement information; (9) object intended use information; or any combination of (1), (2), (3), (4), (5), (6), (7), (8) and (9).

[0145] Clause 15: The method of clause 1, wherein the object is a product sold by a retailer.

[0146] Clause 16: The method of clause 1, wherein the recommendation is presented in an augmented reality manner using 3D graphics.

[0147] Clause 17: The method of clause 1, wherein the recommendations are selected further based on at least one of customer reviews for the object and / or object bounce data.

[0148] Clause 18: An augmented reality (AR) device comprising: a camera; a display; at least one processor; and a memory device storing instructions that, when executed by the at least one processor, cause the AR device to: capture an image stream of a scene using the camera; provide the image stream to a server operating a recommendation engine, wherein the recommendation engine processes the image stream to identify objects and extract contextual information, retrieves a set of recommendations for the objects, and selects a recommendation from the set of recommendations based at least in part on the contextual information; receives the recommendation from the server; and presents the recommendation in AR on the display.

[0149] Clause 19: The AR device of clause 18, further comprising: a global positioning system (GPS) receiver configured to receive global positioning system data to determine a location of the AR device, wherein the contextual information is also based on the location of the AR device.

[0150] Clause 20: The AR device of clause 18, wherein the AR device is one of smart glasses, a smartphone, or a computing tablet.

[0151] Clause 21: A virtual reality (VR) device comprising: a display; at least one processor; and a memory device storing instructions that, when executed by the at least one processor, cause the VR device to: present a virtual scene having an object with an object ID; send the object ID to a server operating a recommendation engine, wherein the recommendation engine extracts context information, retrieves a set of recommendations for the object via the object ID, and selects a recommendation from the set of recommendations based at least in part on the context information; receives the recommendation; and presents the recommendation in VR on the display.

[0152] The various embodiments described above are provided by way of illustration only and should not be construed as limitations on the appended claims. Those skilled in the art will readily appreciate that various modifications and changes may be made without following the example embodiments and applications shown and described herein, and without departing from the true spirit and scope of the appended claims.

Claims

1. A computer-implemented method for guiding a user to interact with an object, the method comprising: receiving an information prompt of a scene including the object, the information prompt being related to data retrieved by the sensor; identifying the object based on the information prompt; extracting context information based on the information prompt; determining at least one interaction indicator configured to indicate an interaction that can be performed with the identified object, the at least one interaction indicator being based at least in part on the contextual information; as well as By controlling a mixed reality (MR) device, a user is guided to perform interaction on the object based on the at least one interaction indicator.

2. The computer-implemented method of claim 1 , wherein: The context information includes one or more of the following: settings; geographic location; life cycle period of the object; environment type; Object defects; object use or misuse; object safety issues; and / or any combination thereof.

3. A computer-implemented method according to any preceding claim, wherein: The context information includes an object life cycle period, the object life cycle period including one of: the object is displayed for sale; the object is during installation; the object is in use; the object is at the end of life, and / or any combination thereof.

4. A computer-implemented method according to any preceding claim, wherein: Guiding the user includes determining a surface of the object, and wherein the at least one interaction indicator is provided attached to the determined surface.

5. The computer-implemented method of claim 4, wherein: The surface is determined by: determining a pose of the object and an unobstructed surface of the object based on information cues as image data and / or mesh data; as well as The surface is determined based on the determined pose of the object and the determined unoccluded surface of the object.

6. The computer-implemented method of claim 5, wherein: The surface is also determined by: A distance to the object and lighting conditions of the scene are determined, wherein determining the surface is further based on the distance to the object and lighting conditions of the scene.

7. A computer-implemented method according to any preceding claim, wherein: Determining at least one interaction indicator includes: determining a plurality of interaction indicators for the identified objects, and The plurality of interaction indicators are ranked, wherein the ranking is based at least in part on the contextual information, and the determined at least one interaction indicator is a highest ranked interaction indicator of the plurality of interaction indicators.

8. The computer-implemented method of claim 7, wherein: Ranking the plurality of interaction indicators includes processing the plurality of interaction indicators using a machine learning model to score the plurality of interaction indicators based at least in part on the contextual information.

9. A computer-implemented method according to any preceding claim, wherein: The at least one interaction indicator comprises an object warning when the extracted context information comprises an identified room type in the scene other than a recommended room type for the object, wherein information regarding the recommended room type for the object is obtained from a database.

10. The computer-implemented method of any preceding claim, further comprising: identifying at least two surfaces of the object; selecting a surface recommendation for each of the at least two surfaces; as well as Surface recommendations are sent to the MR device, wherein each surface recommendation is presented in MR attached to a corresponding surface of the at least two surfaces.

11. The computer-implemented method of claim 10, wherein: Each of the at least two surfaces is made of a different material, and the recommendation selected for each of the at least two surfaces includes care information for each of the different materials.

12. The computer-implemented method of any preceding claim, further comprising: An installation error is identified based at least on the information prompt, the object, and the context information.

13. The computer-implemented method of any preceding claim, further comprising: identifying a surface of the object; comparing a surface of the object to a surface model stored in a database to identify a defective surface; as well as A message is sent to the MR device to provide a recommendation regarding the defective surface.

14. A computer-implemented method according to any preceding claim, wherein: The at least one interaction indicator includes one or more of: object material; object size; object care instructions; disposal information; sustainability information; assembly information; installation error information; consumption / replacement information; object intended use information; and / or any combination thereof.

15. A computer-implemented method according to any preceding claim, wherein: The objects are products sold by retailers.

16. A computer-implemented method according to any preceding claim, wherein: The at least one interaction indicator is presented as a 3D graphic on the MR device.

17. The computer-implemented method of claim 16, wherein: The 3D graphic presented on the MR device is a guiding avatar user who virtually performs the interaction.

18. A computer-implemented method according to any preceding claim, wherein: The at least one interaction indicator is selected based on at least one of customer reviews for the object and / or object bounce data.

19. A computer-implemented method according to any preceding claim, wherein: The information prompt includes one of the following: image data, grid data, auditory data, tactile data, environmental data, motion data, olfactory data, or any combination thereof.

20. A computer-implemented method according to any preceding claim, wherein: The at least one interaction indicator is a visual signal, an auditory signal or a tactile signal.

21. A mixed reality (MR) device, comprising: Camera device; monitor; at least one processor; as well as a memory device storing instructions that, when executed by the at least one processor, cause the MR device to: Get information tips of the scene; processing the information cues to identify the object and extract contextual information; determining at least one interaction indicator configured to indicate an interaction that can be performed with the identified object, the interaction indicator being based at least in part on the contextual information; as well as By controlling the MR device, the user is guided to perform interaction on the object based on the at least one interaction indicator.

22. The MR device according to claim 21, wherein The server is also configured to receive information cues from one of: an image sensor, a depth sensor, an audio sensor, a tactile sensor, a temperature sensor, a humidity sensor, a light sensor, an air quality sensor, an accelerometer, a gyroscope, an inertial measurement unit, an olfactory sensor, or any combination thereof.

23. The MR device according to claim 21 or 22, wherein: The server is further configured to transmit the at least one interaction indicator to a wearable device operably connectable to the MR device.

24. The MR device according to any one of claims 21 to 23, further comprising: A global positioning system (GPS) receiver is configured to receive global positioning system data to determine a location of the MR device, wherein the context information is also based on the location of the MR device.

25. The MR device according to any one of claims 21 to 24, wherein: The MR device is one of smart glasses, a smartphone, or a computing tablet.

26. A server comprising a processing unit, wherein: The processing unit is configured to: Processing information cues received from the MR device to identify the object and extract contextual information; determining at least one interaction indicator configured to indicate an interaction that can be performed with the identified object, the interaction indicator being based at least in part on the contextual information; as well as The at least one interaction indicator is transmitted to the MR device to provide guidance to a user through control of the MR device to perform an interaction with the object based on the at least one interaction indicator.

Images