Method, apparatus and system for delivering recommendations

Abstract

The invention relates to a method, a device and a system for delivering recommendations. An electronic device: obtains unvarnished image data characterizing a field of view captured by an image sensor; determining whether the identified subject in the pass-through image data satisfies a confidence score threshold associated with a user-specific recommended profile; generating one or more computer generated reality (CGR) content items associated with the identified subject in response to determining that the identified subject in the unvarnished image data satisfies a confidence score threshold; and synthesizing the pass-through image data with one or more CGR content items, wherein the one or more CGR content items are proximate to the identified subject in the field of view.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a divisional application of the invention patent application with international application number PCT / US2019 / 050498, international application date September 11, 2019, date of entry into the Chinese national phase February 25, 2021, Chinese national application number 201980056127.6, and invention name “Methods, devices and systems for delivering recommendations”.

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 729,960, filed on September 11, 2018, which is hereby incorporated by reference in its entirety. Technical Field

[0004] The present invention generally relates to delivering recommendations, including but not limited to electronic devices that enable delivery of optimal recommendations in a computer-generated reality environment. Background Art

[0005] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0006] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that a person perceives and / or interacts with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0007] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides a perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0008] Examples of CGR include virtual reality and mixed reality.

[0009] A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment, and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0010] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0011] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0012] Examples of mixed reality include augmented reality and augmented virtuality.

[0013] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0014] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not true versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

[0015] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0016] There are many different types of electronic systems that enable a person to sense and / or interact with various CGR environments. Examples include smartphones, tablets, desktop / laptop computers, head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback and / or cameras with gesture tracking and / or other body pose estimation capabilities).

[0017] The head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, the head-mounted system may be a head-mounted shell (HME) configured to accept an external opaque display (e.g., a smart phone). The head-mounted system may incorporate one or more imaging sensors for capturing images or videos of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, the head-mounted system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light representing the image is directed to the person's eyes. The display may utilize digital light projection, OLED, LED, uLED, silicon-based liquid crystal, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one specific implementation, the transparent or translucent display may be configured to selectively become opaque. A projection-based system may employ retinal projection technology that projects a graphic image onto a person's retina. The projection system may also be configured to project virtual objects into a physical environment, such as a hologram or on a physical surface.

[0018] CGR (AR) technology has the potential to become an integral part of users' daily lives. Devices that implement CGR can provide users with information related to many aspects, from navigation to weather, to architecture, to games, etc. However, the information provided to users may be overwhelming and may not be relevant to the user's interests. Summary of the invention

[0019] According to some embodiments, a method is performed at an electronic device having a non-volatile memory and one or more processors. The method includes obtaining transparent image data representing a field of view captured by an image sensor. The method also includes determining whether the identified subject in the transparent image data meets a confidence score threshold associated with a user-specific recommended profile. The method also includes generating one or more computer-generated reality (AR) content items associated with the identified subject in response to determining that the identified subject in the transparent image data meets the confidence score threshold. The method also includes synthesizing the transparent image data with one or more CGR content items, wherein the one or more CGR content items are close to the identified subject in the field of view.

[0020] According to some embodiments, a method is performed at an electronic device having a non-volatile memory and one or more processors. The method includes obtaining a first set of subjects associated with a first posture of the device. The method also includes determining a likelihood estimate for each of the first set of subjects based on a user context and the first posture. The method also includes determining whether at least one likelihood estimate for at least one corresponding subject in the first set of subjects exceeds a confidence threshold. The method also includes, in response to determining that at least one likelihood estimate exceeds a confidence threshold, generating recommended content or actions associated with at least one corresponding subject using a user context and at least one classifier associated with the at least one corresponding subject.

[0021] According to some embodiments, the electronic device includes a display, one or more input devices, one or more processors, a non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the execution of the operations of any of the methods described herein. According to some embodiments, a non-volatile computer-readable storage medium has instructions stored therein, which, when executed by one or more processors of an electronic device having a display and one or more input devices, cause the device to perform or cause the execution of the operations of any of the methods described herein. According to some embodiments, the electronic device includes: a display, one or more input devices; and a device for performing or causing the execution of the operations of any of the methods described herein. According to some embodiments, an information processing device used in an electronic device having a display and one or more input devices includes a device for performing or causing the execution of the operations of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0023] Figure 1 is a block diagram of an exemplary operating environment according to some implementations.

[0024] Figures 2A to 2G An exemplary user interface for rendering user-specific computer-generated reality (CGR) content items is shown according to some embodiments.

[0025] Figure 3 An exemplary abstract block diagram for generating user-specific CGR content according to some embodiments is shown.

[0026] FIG. 4A to FIG. 4C An exemplary user interface for recommending user-specific CGR content items based on updated user context and / or gestures is shown in accordance with some embodiments.

[0027] Figure 5 An exemplary abstract block diagram for delivering optimal recommendations in a CGR environment is shown according to some embodiments.

[0028] Figure 6 A flowchart of a method for rendering user-specific CGR content items according to some embodiments is shown.

[0029] Figure 7 A flow chart of a method of generating recommended CGR content according to some embodiments is shown.

[0030] Figure 8 A block diagram of a computing device according to some embodiments. DETAILED DESCRIPTION

[0031] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0032] In the embodiments described below, the transparent image data representing the field of view captured by the image sensor is synthesized with one or more computer generated reality (CGR) content items. One or more CGR content items are associated with the identified subject in the transparent image data, and the identified subject in the transparent image data satisfies a confidence score threshold. In the synthetic image, one or more CGR content items are placed close to the identified subject in the field of view. Therefore, the embodiments described below provide seamless integration of user-specific content. User-specific content is generated based on the likelihood of user interest and displayed to the user. For example, when a cupcake is identified in the user's field of view, a cupcake recipe or nutritional information for the cupcake is generated and displayed to the user. Therefore, the recommended CGR content items generated according to the various embodiments described herein allow the user to remain immersed in his experience without having to manually enter a search query or indicate a preference. Seamless integration also reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0033] In the following embodiments, a set of subjects associated with the posture of the device is obtained, and a likelihood estimate of each of the set of subjects is determined based on the user context and the posture. Recommended content or action associated with at least one corresponding subject in the set of subjects is generated. In response to determining that at least one likelihood estimate of at least one corresponding subject in the set of subjects exceeds a confidence threshold, recommended content or action is generated using at least one classifier associated with at least one corresponding subject. Therefore, the embodiments described below provide a process for generating recommended CGR content based on the degree of likelihood that the user will be interested in the subject. Therefore, content recommendations according to the various embodiments described herein provide a seamless user experience that requires less time and user input when locating information or the next action. This also reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0034] Figure 1 1 is a block diagram of an exemplary operating environment 100 according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 102 and a CGR device 104. In Figure 1 In the example of , the CGR device 104 is worn by the user 10 .

[0035] In some implementations, the CGR device 104 corresponds to a tablet or mobile phone. In various implementations, the CGR device 104 corresponds to a head-mounted system, such as a head-mounted device (HMD) or a head-mounted housing (HME) into which a tablet or mobile phone is inserted. In some implementations, the CGR device 104 is configured to present CGR content to a user. In some implementations, the CGR device 104 includes a suitable combination of software, firmware, and / or hardware.

[0036] According to some implementations, the CGR device 104 presents CGR content to the user via the display 122 while the user is virtually and / or physically present within the scene 106. In some implementations, the CGR device 104 is configured to present virtual content (e.g., the virtual cylinder 109) and enable video pass-through of the scene 106 (e.g., including the representation 117 of the table 107) on the display. In some implementations, the CGR device 104 is configured to present virtual content and enable optical see-through of the scene 106.

[0037] In some implementations, the user holds the CGR device 104 in his / her hand. In some implementations, the user wears the CGR device 104 on his / her head. Thus, the CGR device 104 includes one or more CGR displays configured to display CGR content. For example, the CGR device 104 surrounds the user's field of view. In some implementations, the CGR device 104 is replaced with a CGR cavity, housing, or chamber configured to present CGR content, wherein the user no longer wears the CGR device 104.

[0038] In some implementations, the controller 102 is configured to manage and coordinate the presentation of CGR content for the user. In some implementations, the controller 102 includes a suitable combination of software, firmware and / or hardware. In some implementations, the controller 102 is a computing device that is locally or remotely located relative to the scene 106. For example, the controller 102 is a local server located within the scene 106. In another example, the controller 102 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 106. In some implementations, the controller 102 is communicatively coupled to the CGR device 104 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE802.11x, IEEE802.16x, IEEE802.3x, etc.). In some implementations, the functionality of the controller 102 is provided by the CGR device 104 and / or combined with the CGR device.

[0039] like Figure 1As shown, the CGR device 104 presents a representation of the scene 106. In some implementations, the representation of the scene 106 is generated by the controller 102 and / or the CGR device 104. In some implementations, the representation of the scene 106 includes a virtual scene that is a simulated replacement of the scene 106. In other words, in some implementations, the representation of the scene 106 is simulated by the controller 102 and / or the CGR device 104. In such implementations, the representation of the scene 106 is different from the scene 106 in which the CGR device 104 is located. In some implementations, the representation of the scene 106 includes an augmented scene that is a modified version of the scene 106 (e.g., including the virtual cylinder 109). For example, in some implementations, the controller 102 and / or the CGR device 104 modifies (e.g., augments) the scene 106 in order to generate the representation of the scene 106. In some implementations, the controller 102 and / or the CGR device 104 generates the representation of the scene 106 by simulating a replica of the scene 106. In some implementations, the controller 102 and / or the CGR device 104 generates a representation of the scene 106 by removing and / or adding items to a simulated copy of the scene 106 .

[0040] Figures 2A to 2G An exemplary user interface for rendering user-specific computer generated reality (CGR) content according to some embodiments is shown. The user interface in these figures is used to illustrate the Figure 5 In some embodiments, the device detects input via an input device separate from the display (e.g., a head mounted device (HMD) with voice activated commands, a laptop with a separate touchpad and display, or a desktop computer with a separate mouse and display).

[0041] like Figure 2A As shown, device 104 displays a media capture / interaction interface 202. According to some embodiments, the media capture / interaction interface 202 displays a scene with a subject in the field of view of the image sensor. Image data (or transparent image data) representing the scene is captured by the image sensor. In some embodiments, the transparent image data includes a preview image, a surface image (e.g., a flat surface), a depth map, anchor coordinates (e.g., for depth mapping), etc. In some embodiments, the transparent image data includes not only visual content, but also audio content, 3D rendering, timestamps (of the actual frame displayed), header files (e.g., camera settings such as contrast, saturation, white balance, etc.), and / or metadata.

[0042] As mentioned above Figure 1As described above, in some embodiments, the image sensor used to capture the scene is part of or attached to device 104; while in some other embodiments, the image sensor is separate from device 104, such as on a camera remote from device 104. In various embodiments, the scene changes as the field of view of the image sensor changes, as described below with reference to Figure 2C to Figure 2G As shown. Figure 2A In FIG. 2 , the media capture / interaction interface 202 includes an open doorway with a door symbol 210 labeled “201.” The media capture / interaction interface 202 also shows a picture frame 220 and a table 230 in the room through the open doorway.

[0043] Figure 2B Synthetic pass-through image data with CGR content items rendered in the media capture / interaction interface 202 is shown. According to some embodiments, the synthetic pass-through image data includes information, such as room information 212 and floor map 214 associated with the room. The room information 212 and floor map 214 are CGR content items generated based on the device 104 recognizing the door sign 210 and determining that the user is interested in learning more about the room and the building. In some embodiments, as indicated by the dotted line around the door sign 210, the identified subject in the field of view is emphasized to indicate the association of the additional CGR content items 212 and 214 with the identified subject 210. In some embodiments, the CGR content items 212 and 214 are displayed in an animated manner (e.g., flashing, shrinking / enlarging, moving, etc.) near the identified subject 210 to indicate the association with the identified subject 210. In some embodiments, in addition to or instead of displaying CGR content items 212 and 214 on the display of device 104, audio content is played as the CGR content items, for example, reading door signs, room information, and / or floor maps to the user.

[0044] Figure 2B to Figure 2C 2 shows a sequence in which the media capture / interaction interface 202 is updated based on a change in the field of view of the image sensor. The viewing angle or vantage point of the image sensor is Figure 2B to Figure 2C For example, in Figure 2C , the doorway is no longer displayed in the media capture / interaction interface 202, indicating that the user has entered the room. Figure 2B The CGR content items 212 and 214 shown associated with the door sign 210 are no longer provided to the user. Instead, when the user enters the room, the user has a complete view of the room. Figure 2C As shown, the media capture / interaction interface 202 displays three walls of the room. The media capture / interaction interface 202 also displays a picture frame 220, a table 230, a clock 240, and a dog 236 in the room. Figure 2CAs shown, media capture / interaction interface 202 displays cupcake 232 and book 234 on table 230 .

[0045] FIG. 2D to FIG. 2E Different CGR content items are shown rendered to the user based on different user contexts. Figure 2D , the synthetic pass-through image data includes a CGR content item 250 associated with a cupcake 232. The CGR content item 250 is rendered adjacent to or relative to the cupcake 232. In addition, the CGR content item 250 includes information associated with the cupcake 232, such as the calories of the cupcake, and an affordance including a link 252 to a recipe for the cupcake 232 and a button 254 for adding the cupcake 232 to a food diary. The affordance 252 is provided to the user as an option to perform an action associated with the cupcake 232, such as tapping the link 252 to find a recipe for the cupcake 232 or clicking the button 254 to add the cupcake 232 to a food diary.

[0046] Figure 2D The CGR content item 250 shown in FIG. 1 is generated based on determining that the user is interested in the cupcake 232, and a recommendation is made to provide information about the cupcake 232. In contrast, Figure 2E Different CGR content items 256 are shown overlaid on the cupcake 232. While the user remains interested in the cupcake 232, the CGR content items 256 proceed based on different user contexts (e.g., the user has dietary restrictions, etc.).

[0047] Figure 2F A CGR content item 260 is shown proximate to an identified subject (e.g., table 230), wherein the CGR content item 260 is generated in response to detecting a gaze proximate to an area 262 containing at least a portion of the identified subject 230. Figure 2F , device 104 detects region 262 proximate to gaze, which includes a portion of table 230, a portion of cupcake 232 on table 230, and a portion of book 234 on table 230. Device 104 identifies table 230 using a subset of the pass-through image data corresponding to region 262 and applying a table classifier to the subset of the image data.

[0048] In some embodiments, a table classifier is selected based on weights assigned to a cluster of classifiers. In some embodiments, a classifier corresponds to an entry in a library of objects / subjects, e.g., shape, quantity, animal, food, plant, person, dog, square, flower, shape, lighting, etc. Using one or more classifiers, a subject can be identified in the image data. During subject identification, weights are assigned to different classifiers, and one or more classifiers can be selected based on the weights associated with each classifier. The selected classifier can then be used to identify the subject in the image data.

[0049] For example, based on gaze near region 262, weights are assigned to a table classifier, a cupcake classifier, and a book classifier. As gaze stabilizes on the table surface, the weight assigned to the table classifier increases, while the weights assigned to the cupcake classifier and the book classifier decrease. Based on the weights assigned to the classifiers, a table classifier is selected for identifying a table subject 230 near gaze region 262. Having identified table 230, device 104 renders CGR content 260 adjacent to table 230, such as a recommendation for a chair that may match the style of table 230.

[0050] Figure 2G A CGR content item 270 (e.g., a hand icon in a pointing configuration) is shown proximate to the identified subject 234, with a gaze area 272 within a threshold distance from the identified subject 234. Figure 2G , the device 104 detects that the gaze area 272 is on the dog 236 in the field of view. However, based on the user context, the user is unlikely to be interested in seeing more information about the dog 236 displayed in the media capture / interaction interface 202, for example, the user is afraid of animals. Further based on the user context, the device determines that the user is more interested in the book 234 (e.g., the user recently obtained the book 234 from the library), and the book 234 is within a threshold distance from the gaze area 272. Subsequently, the device 104 expands the gaze area 272 so that more subjects are included in the area and analyzed. The book 234 is then identified from the image data corresponding to the expanded gaze area, and the CGR content item 270 is generated and rendered above the book 234.

[0051] Figure 2G The CGR content item 270 is shown to be generated for a particular user by likelihood estimation, where prior information about the user and the current pass-through image data are input. This is particularly useful when the identified subject includes multiple searchable elements and each element is associated with at least a classifier. For example, in Figures 2A to 2GIn the example, the picture frame 220 includes multiple searchable elements, including the frame itself, the vase in the picture, and the flowers in the vase. In order to distinguish these searchable elements and generate CGR content items for the elements that the user is most likely to be interested in, as described below with reference to Figure 3 Described in more detail, fine-tuning content recommendations.

[0052] Figure 3 An abstract block diagram associated with a multiple iterative process 300 for identifying subjects that are most likely to be of interest to a user is shown. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein. To this end, as a non-limiting example, in Figure 3 In FIG. 2 , since the fixation region 222 is close to the picture frame 220 in the field of view, the picture frame 220 includes a plurality of searchable elements, including a frame 310, a flower 320, and a vase 330, and each of the searchable elements is close to the fixation region. The likelihood estimate is determined during multiple iterations.

[0053] In some embodiments, each likelihood estimate is assigned an initial value, for example, all likelihood estimates are 0 or the likelihood estimates are uniformly distributed. Figure 3 As shown in , during the first iteration, the likelihood estimates for frame 310, flower 320, and vase 330 are equally distributed to approximately 1 / 3, for example, frame 310 is 0.33, flower 320 is 0.33, and vase 330 is 0.34. During the second iteration, the likelihood estimates are updated to reflect what the user is interested in at the next time step after the first iteration. In addition, as will be referred to below FIG. 4A to FIG. 4C As described in detail, changes in posture and / or user context can contribute to changes in likelihood estimates. Thus, during the second iteration, the likelihood estimate for frame 310 is 0.25, the likelihood estimate for flower 320 is 0.00, and the likelihood estimate for vase 330 is 0.75. Similarly, during the third iteration, further changes in posture and / or user context cause the likelihood estimates for frame 310 to change to 0.75, flower 320 to 0.00, and vase 330 to 0.25. Although the likelihood estimate of 0.00 for flower 320 indicates that the user is most likely not interested in learning more about flower 320, in some embodiments, the device will require more iterations to identify the one element that the user is most interested in, e.g., values ​​of 0.25 and 0.75 do not exceed the confidence threshold. As Figure 3 As shown in , during the fourth and final iteration, the likelihood estimate for frame 310 has increased to 0.90, indicating that the user is most likely interested in the frame itself, rather than the picture depicted in the frame.

[0054] Figure 3The selection process shown in is funnel-shaped so that over time, for example, after the second and third iterations or a threshold amount of time, likelihood estimates below a threshold (e.g., flowers with a likelihood estimate of 0.00) are not included in the next iteration. After multiple iterations, the likelihood estimates converge to a particular value so that recommendations can be made for a particular subject that the user is most likely to be interested in.

[0055] Go to FIG. 4A to FIG. 4C , FIG. 4A to FIG. 4C An exemplary user interface for rendering user-specific CGR content items based on user context and / or gesture according to some embodiments is shown. Figure 5 The recommended content generation process in .

[0056] For example, in Figure 4A , device 104 detects a gaze area 222 (shown as a dashed line) proximate to a picture frame 220 based on the posture of device 104. Picture frame 220 includes the frame itself, the vase in the picture, and the flowers in the vase. A likelihood estimator of device 104 determines a likelihood estimate for each subject, such as a likelihood estimate for the frame, a likelihood estimate for the vase, and a likelihood estimate for the flowers. In some embodiments, the likelihood estimates are determined based on both the user context and the posture. Figure 4A , the gaze area 222a is close to the frame, the vase, and the flowers. Using the user context, for example, if the user is a botanist rather than an artist, the user is more likely to be interested in the flowers depicted in the frame 220. Therefore, the device 104 generates the recommended content 224 to provide the user with information about the flowers.

[0057] FIG. 4B to FIG. 4C Shows relative to Figure 4A The interface shown in FIG. updates the media capture / interaction interface 202. Figure 4A The view shown, FIG. 4B to FIG. 4C The viewing angle or vantage point of the device 104 is shown changing as the field of view shifts to the right (eg, due to movement of the device 104). As the gaze shifts to the right, the gaze area 222b moves away from the center picture frame 220 and to the right. Figure 4B It is shown that as a result of the gesture change, the device 104 predicts that the clock on the right wall is the subject of interest to the user, and generates an event calendar 242 adjacent to the clock 240 . Figure 4C It is shown that when the gaze area 222c moves even closer to the clock 240, another recommended content 244 is shown in the interface 202. The recommended content 244 is generated based on the user context that it is time for a veterinary visit and the user is more interested in obtaining information associated with the dog 236 to prepare for the veterinary visit.

[0058] Figure 5An abstract block diagram associated with a process 500 for delivering optimal recommendations in a CGR environment is shown, according to some embodiments. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein. For example, a system implementing process 500 can be a heterogeneous system including multiple distributed devices. Thus, as indicated by the dashed lines, some components that perform computationally resource-intensive tasks are implemented on remote agents and can be dynamically reconfigured between local agents, peer agents, and distributed agents.

[0059] like Figure 5 As shown, a scanner 510 (e.g., Figure 8 The image capture control module 850 in the scanner 510 obtains an image and exports image data or transparent image data. In some embodiments, the transparent image data includes audio content, visual content, 3D rendering, timestamp (of the actual frame displayed), header file (containing all camera settings: contrast, saturation, white balance, etc.) and / or metadata. In some embodiments, the transparent image data includes preview images, surface images (e.g., flat surfaces), depth maps, anchor coordinates (e.g., for depth mapping). In some embodiments, the scanner 510 also provides posture information of the device together with the transparent image data, such as the focus within the field of view of the image sensor, the distance from the image sensor to multiple real-world objects, the percentage of visual space occupied by the subject in the field of view, and / or the current gaze, etc.

[0060] In some embodiments, the user context is specified in a user-specific recommendation profile 505. In some embodiments, the user-specific recommendation profile includes user history, user-specific lists, user-enabled modules (e.g., occupation-specific or task-specific, such as engine repair), etc.

[0061] In some embodiments, the analyzer 520 includes a plurality of classifiers 522. In some embodiments, the plurality of classifiers 522 correspond to entries in a subject library, such as shape, quantity, animal, food, plant, person, etc. The classifiers are provided to the likelihood estimator 530 along with associated weights (e.g., a dog classifier for identifying dogs, etc.).

[0062] Along with input from analyzer 520, likelihood estimator 530 receives image data and pose information from scanner 510 and receives user context 505. Based on the received information, likelihood estimator 530 identifies objects in the field of view that the user is most likely interested in, and generates recommended CGR content items 560 for the user to view and / or interact with, such as Figures 2A to 2G and FIG. 4A to FIG. 4C shown.

[0063] In some embodiments, cascade caches 550-1, 550-2, 550-3, ... 550-N are used to facilitate subject recognition and CGR content item recommendations. Subjects and associated recommendations are stored in the cascade caches in weight order. For example, during one iteration, the first cascade cache 550-1 stores the subject with the lowest recommendation weight, and the last cascade cache 550-N stores the subject with the highest recommendation weight. Therefore, the first cascade cache 550-1 includes information about the subjects determined to be the least important or relevant to the user at that stage, and the last cascade cache 550-N includes information about the subjects determined to be the most important or relevant to the user at that stage. In the example Figure 3 During subsequent stages or iterations shown in , the information stored in cascaded cache 550 may be adjusted based on user context and / or posture changes.

[0064] In some embodiments, precise matching 540 is performed to fine-tune the results from likelihood estimator 530. In some embodiments, precise matching 540 is performed remotely (e.g., at a second device) to save computing resources of a local device. In such embodiments, encoder 532 is used to reduce vector dimensions to efficiently transmit data to a remote source. Upon receiving the encoded data, a decoder 542 on the remote source decodes the data before performing fine-grained matching. In some embodiments, at a remote source, machine learning is applied across multiple users so that better recommendations can be generated for a particular user.

[0065] Figure 6 6 is a flowchart representation of a method 600 for rendering a user-specific CGR content item according to some embodiments. In some embodiments (and as detailed below as an example), the method 600 is performed by an electronic device (or a portion thereof), such as Figure 1 The electronic device 104 or Figure 3300 in the device, which includes a non-transitory memory and one or more processors. In some embodiments, the device also includes an image sensor or camera assembly, a display, and one or more input devices. For example, the display and one or more input devices are combined into a touch screen display. In this example, the electronic device corresponds to a smart phone or a tablet computer. As another example, the display and one or more input devices are separated. In this example, the electronic device corresponds to a laptop computer or a desktop computer. For example, the electronic device corresponds to a wearable computing device (including an HMD that surrounds or does not surround the user's eyes or a CGR rendering device with one or more CGR displays), a smart phone, a tablet computer, a laptop computer, a desktop computer, a self-service machine, a set-top box (STB), a set-top box (OTT), a game console, etc. In some embodiments, the image sensor is separated from the device, for example, on a camera away from the device 104.

[0066] In some embodiments, method 600 is performed by a processing logic component (including hardware, firmware, software, or a suitable combination thereof). In some embodiments, method 600 is performed by one or more processors that execute codes, programs, or instructions stored in a non-transitory computer-readable storage medium (e.g., a non-transitory memory). Some operations in method 600 are optionally combined, and / or the order of some operations is optionally changed. In short, the method 600 includes: obtaining transparent image data representing a field of view captured by an image sensor. Determine whether the identified subject in the transparent image data satisfies a confidence score threshold associated with a user-specific recommended profile; in response to determining that the identified subject in the transparent image data satisfies the confidence score threshold, generate one or more computer-generated reality (CGR) content items associated with the identified subject; and synthesize the transparent image data with one or more CGR content items, wherein the one or more CGR content items are close to the identified subject in the field of view.

[0067] At block 602, method 600 begins, where an electronic device obtains scene data. According to some embodiments, device 104 or a component thereof (e.g., Figure 8 The image capture control module 850 in the device 104 obtains scene data (e.g., image data or pass-through image data) by capturing an image of the scene from a first reference / vantage point (e.g., camera position, pose, or field of view) with an image sensor or camera assembly. The device 104 or a component thereof (e.g., Figure 8 The image processing module 852 in the embodiment 1 derives the transparent image data representing the field of view. Figure 2AIn the example, the device 104 obtains the transparent image data and displays the media capture / interaction interface 202, which includes a scene corresponding to the room, which has a door sign 210 on the door of the room, a table 230, and a picture frame 220 in the room. Figure 2C , the media capture / interaction interface 202 depicts a scene within a room, which also includes a clock 240 on the right wall, a dog 236 near the left wall, and a cupcake 232 and a book 234 on a table 230 .

[0068] Method 600 continues at block 604, where the electronic device determines whether the identified subject in the pass-through image data satisfies a confidence score threshold associated with a user-specific recommendation profile. In other words, device 104 or a component thereof (e.g., Figure 8 The subject recognition module 854 or Figure 5 In some embodiments, the electronic device obtains information about the user's preferences based on a recommendation profile specific to the user. For example, referring to Figure 2A , device 104 uses user-specific recommendation profiles such as user history, user-specific lists, user-enabled modules (e.g., occupation-specific or task-specific, such as engine repair), etc. to determine whether the user is interested in learning more about the room and building associated with door sign 210.

[0069] In some embodiments, the user-specific recommendation profile includes at least one of the context of the user interacting with the device, the user's biometrics, the user's previous searches, or the user's profile. For example, the context of the user interacting with the device includes orders recently placed by the user from a veterinarian, a cupcake baker, etc. In another example, a biometric sensor may be used to measure a biometric of the user, such as elevated blood pressure and / or heart rate indicating sadness or excitement experienced by the user for a subject. In yet another example, the user-specific recommendation profile includes the user's previous searches and associated actions taken, such as the user having searched for cupcakes multiple times before but having decided to say "no" to cupcakes on all previous occasions. In yet another example, metadata in the user profile may display prior information used to assign weights and / or likelihood estimates.

[0070] In some embodiments, as shown in block 606, an identified subject in the pass-through image data is identified by detecting a gaze at an area in the field of view, as shown in block 608, a subset of the pass-through image data corresponding to the area is obtained, and as presented in block 610, the identified subject is identified based on the subset of the pass-through image data and a classifier. Figure 2BIn the embodiment of the present invention, upon detecting a gaze at an area near the door sign 210, the device 104 or a component thereof (e.g., Figure 8 The image processing module 852 or Figure 5 The likelihood estimator 530 in the method further processes the subset of the transparent image data corresponding to the area near the door sign 210. Then, the device 104 uses the subset of the transparent image data and the door sign classifier to recognize the door sign 210.

[0071] In some embodiments, method 600 further continues at block 612, where the electronic device assigns weights to classifiers based on the gaze, where each of the classifiers is associated with a subject in the gaze area, and the weights of the classifiers are adjusted based on updates to the gaze. In some embodiments, method 600 further continues at block 614, where the electronic device selects a classifier with the highest weight from the classifiers.

[0072] For example, in Figure 2C , before the gaze is detected, equal weights are assigned to all objects in the field of view, for example, equal weights are assigned to the picture frame 220, the table 230, the cupcake 232, the book 234, the clock 240, and the dog 236. Figure 2D and Figure 2E , once the gaze is stabilized on the cupcake 232 on top of the table 230, the weight associated with the cupcake classifier increases, while the weights associated with the other classifiers decrease. Therefore, the cupcake classifier is selected from the classifiers in order to identify the cupcake 232 subject and recommend CGR content items associated with the cupcake 232, for example, Figure 2D , or as shown in FIG. 250 with a link 252 to a cupcake recipe and an add affordance (e.g., button 254). Figure 2E No cupcake logo is shown (eg, CGR content item 256 ).

[0073] In some embodiments, as shown in block 616, the gaze area includes at least a portion of the identified subject. Figure 2F , gaze area 262 includes a portion of table 230, a portion of cupcake 232 on table 230, and a portion of book 234 on table 230. Device 104 identifies table 230 using at least a portion of the table image data and applying a table classifier to the table image data to identify table 230. In contrast, in some embodiments, as shown in block 618, the gaze area is within a threshold distance from the identified subject recommended by the CGR content, and the identified subject is identified based on a recommendation profile specific to the user. For example, in Figure 2G, gaze region 272 is near dog 236, and the identified subject is book 234 on table 230. Book 234 is identified as the subject that the user is most likely interested in because the user-specific recommendation profile indicates that the user is more interested in book 234 than dog 236, and book 234 is within a threshold distance from gaze region 272. Thus, gaze region 272 is expanded to include the book image data, a higher weight is assigned to the book classifier, and the book classifier is used to process the expanded image data to identify book 234 as the subject of interest.

[0074] In some embodiments, as shown in block 620, the identified subject includes multiple searchable elements, and each element is associated with at least one classifier. For example, the picture frame 220 includes multiple searchable elements, the frame itself, the vase in the picture, and the flowers in the vase. In order to distinguish these searchable elements and generate CGR content items for the elements that the user is most likely to be interested in, as described above with reference to Figure 3 Described, fine-tuning content recommendations.

[0075] Still reference Figure 6 , method 600 continues at block 622, where the electronic device generates one or more computer-generated reality (CGR) content items associated with the identified subject in response to determining that the identified subject in the pass-through image data satisfies the confidence score threshold. According to some embodiments, the device 104 or a component thereof (e.g., Figure 8 The one or more CGR content items generated by the CGR content recommendation module 856 in the example include at least one of information associated with the identified subject or an option to perform an action associated with the identified subject. Figure 2B The text 212 about the room and the text 214 about the floor shown in FIG. Figure 2D , a link 252 to a cupcake recipe, and a button 254 for adding the cupcake to the food diary, Figure 2E The cupcake-free symbol in (e.g., CGR content item 256), such as Figure 2F The chair recommendations shown in (e.g., CGR content item 260) or as Figure 2G An indicator 270 pointing to the book 234 is shown in FIG.

[0076] Method 600 continues at box 624, where the electronic device synthesizes the transparent image data with one or more CGR content items. In some embodiments, the electronic device further renders the transparent image data in the field of view as having one or more CGR content items displayed near the identified subject. In some other embodiments, one or more CGR content items are displayed adjacent to the identified subject according to the field of view of the user using the device. For example, in the case of glasses that support CGR, a camera with an image sensor and the user's optical system may be two independent things. Therefore, the position of one or more CGR content items can be determined based on the image sensor or the user's field of view. Alternatively, the image sensor and the user's field of view can be coordinated, for example, one can cover the other. In such embodiments, the position of one or more CGR content items can be determined based on the image sensor and the user's field of view.

[0077] For example, device 104 or a component thereof (e.g., Figure 8 CGR content rendering module 858 or Figure 5 The view and interaction module 560 in the embodiment displays text or logo about the subject next to the subject, for example, Figure 2A As shown, room information 212 and floor map 214 are displayed next to door sign 210. Figure 2E As shown, the no cupcake logo 256 is superimposed on the cupcake 232, such as Figure 2F As shown, a chair recommendation (e.g., CGR content item 260) is displayed next to table 230, and as shown Figure 2G The pointing mark 270 is shown floating to the book 234. In another example, the device 104 or a component thereof (e.g., Figure 8 CGR content rendering module 858 or Figure 5 The view and interaction module 560 in the example displays a link to the subject adjacent to the subject, for example, Figure 2D A link 252 to a cupcake recipe is shown displayed above the cupcake 232. In yet another example, the device 104 or a component thereof (e.g., Figure 8 CGR content rendering module 858 or Figure 5 The view and interaction module 560 in the adjacent subject displays an interactive affordance representation, for example, Figure 2D Button 254 is shown displayed next to cupcake 232 .

[0078] Figure 7 is a flowchart representation of a method 700 for generating recommended CGR content according to some embodiments. In some embodiments (and as detailed below as an example), the method 700 is performed by an electronic device (or a portion thereof), such as Figure 1 The electronic device 104 or Figure 3300 in the device, which includes a non-transitory memory and one or more processors. In some embodiments, the device also includes an image sensor or camera assembly, a display, and one or more input devices. For example, the display and one or more input devices are combined into a touch screen display. In this example, the electronic device corresponds to a smart phone or a tablet computer. As another example, the display and one or more input devices are separated. In this example, the electronic device corresponds to a laptop computer or a desktop computer. For example, the electronic device corresponds to a wearable computing device (including an HMD that surrounds or does not surround the user's eyes or a CGR rendering device with one or more CGR displays), a smart phone, a tablet computer, a laptop computer, a desktop computer, a self-service machine, a set-top box (STB), a set-top box (OTT), a game console, etc. In some embodiments, the image sensor is separated from the device, for example, on a camera away from the device 104.

[0079] In some embodiments, method 700 is performed by a processing logic component (including hardware, firmware, software, or a suitable combination thereof). In some embodiments, method 700 is performed by one or more processors that execute code, programs, or instructions stored in a non-transitory computer-readable storage medium (e.g., a non-transitory memory). Some operations in method 700 are optionally combined, and / or the order of some operations is optionally changed. In short, method 700 includes: obtaining a first group of subjects associated with a first posture of the device; determining a likelihood estimate for each of the first group of subjects based on a user context and a first posture; determining whether at least one likelihood estimate of at least one corresponding subject in the first group of subjects exceeds a confidence threshold; and in response to determining that at least one likelihood estimate exceeds a confidence threshold, generating recommended content or actions associated with at least one corresponding subject using a user context and at least one classifier associated with at least one corresponding subject.

[0080] Method 700 begins at block 702, where the electronic device obtains a first set of subjects associated with a first posture of the device. According to some embodiments, device 104 or a component thereof (e.g., Figure 8 The image capture control module 850 or Figure 5 The scanner 510 in the example embodiment of the present invention obtains scene data (e.g., image data or pass-through image data) by capturing an image of the scene from a first reference / vantage point (e.g., camera position, pose, or field of view) with an image sensor or camera assembly. The device 104 or a component thereof (e.g., Figure 8 In some embodiments, the image processing module 852 in the image processing module 852 derives image data representing the field of view and identifies a set of objects in the field of view. In some embodiments, the identification is performed by (e.g., by the device 104 or its components, such as Figure 8The subject recognition module 854 in the first group of subjects: detecting gazes of a first area in the field of view of the near device, obtaining image data corresponding to the first area, and classifying the first group of subjects based on the image data and one or more classifiers, as described above with reference to Figure 6 Explained.

[0081] At block 704, method 700 continues with the electronic device determining a likelihood estimate for each of the first set of subjects based on the user context and the first gesture. Figure 5 and Figure 4A As shown, the device 104 or a component thereof (e.g., Figure 8 CGR content recommendation module 856 or Figure 5 The likelihood estimator 530 in derives a set of objects near the fixation area 222, including the frame itself 310, the flower 320, and the vase 330. Figure 3 As shown, the device determines likelihood estimates for frame 310, flower 320, and vase 330. In some embodiments, the likelihood estimates correspond to a magnitude / weight of the likelihood that the user is interested in each of the plurality of subjects.

[0082] In some embodiments, the likelihood estimate is determined recursively. As shown in block 706, in some embodiments, the likelihood estimate is determined recursively based on the user context updated during multiple time periods. For example, Figure 3 , the likelihood estimate value during the first iteration is assigned during the first time period, and the value of the user context may be updated during the second time period between the first iteration and the second iteration. Thus, the likelihood estimates for frame 310, flower 320, and vase 330 are updated based on the updated value of the user context (e.g., the user is no longer interested in flower 320). As shown in block 708, in some embodiments, the likelihood estimate value is recursively determined based on the updated gesture. In some embodiments, device 104 or a component thereof (e.g., Figure 8 The image capture control module 850 and / or Figure 8 The image processing module 852 or Figure 5 Scanner 510 in the apparatus) obtains a second set of subjects associated with a second posture of the device, wherein at least one subject is in the first set and the second set of subjects, and determines at least one likelihood estimate of the at least one subject based on the second posture, the user context, and the first posture.

[0083] For example, in FIG. 2A to FIG. 2B In the example, device 104 obtains transparent image data from the first reference point before entering the room. FIG. 2A to FIG. 2B The scene shown includes objects such as a door sign 210 on a room door, a table 230, and a picture frame 220 within the room. Figure 2C, the reference point has changed as the user enters the room, where within the room, media capture / interaction interface 202 depicts a scene including subjects such as clock 240 on the right wall, dog 236 near the left wall, and cupcake 232 and book 234 on table 230. Thus, due to the change in reference point, the likelihood that the user will see more information about the room or building associated with the door sign changes. Figure 4A In , device 104 obtains the scene with picture frame 220 at the center of the field of view. Figure 4B In the example, the posture change causes the field of view to shift from viewing the picture frame 220 in the center to viewing more of the clock 240 hanging on the right wall. Therefore, the likelihood estimates of the picture frame 220 and the clock 240 change.

[0084] In some embodiments, the likelihood estimates are assigned initial likelihood estimates (e.g., all likelihood estimates are 0) or the likelihood estimates are uniformly distributed (e.g., frame 310, flower 320, and vase 330 are initially assigned equal values, such as Figure 3 ). In some embodiments, the initial likelihood estimate is determined by the user context. In one example, an electrical engineer indicates that he is interested in using an "Electrical Engineer Book," where the Electrical Engineer Book contains a cluster of classifiers that include topics specific to electrical engineering (e.g., signal processing, welding, control systems, etc.). The Electrical Engineer Book may also contain a corresponding initial likelihood estimate for each topic. For example, signal processing will have a higher likelihood estimate than mechanics. In some embodiments, the initial likelihood estimate is determined by the first posture information (e.g., what is currently in the field of view of the image sensor), the percentage of visual space occupied by the subject in the field of view (e.g., a whiteboard occupies more space than a dry erase marker), the distance of the subject to the image sensor and / or the current gaze, etc. For example, in Figure 2C In , as the user enters the room, cupcake 232 may have a higher initial likelihood estimate than picture frame 220 due to its proximity to the door. Figure 2C , picture frame 220 may have a higher initial likelihood estimate than cupcake 232 due to occupying a higher percentage of visual space.

[0085] Method 700 continues at block 710, where the electronic device determines whether at least one likelihood estimate for at least one corresponding subject in the first set of subjects exceeds a confidence threshold. For example, given subjects A, B, and C (where the likelihood estimates are A=0.4, B=0.3, and C=0.3), device 104 and components thereof (e.g., Figure 8 CGR content recommendation module 856 or Figure 5The likelihood estimator 530 in ) determines that subject A exceeds the confidence threshold of 0.3. In some embodiments, as shown in block 712, the determination includes a comparison with other likelihood estimates. For example, given subjects A, B, and C, where the likelihood estimates are A=0.6, B=0.2, and C=0.2, the likelihood estimate for subject A>the likelihood estimate for (B+C), so subject A exceeds the confidence threshold.

[0086] In some embodiments, none of the likelihood estimates exceeds the threshold, or multiple likelihood estimates exceed the threshold together. In such embodiments, more than one iteration is required to recursively determine updated likelihood estimates, as described above with reference to steps 706 and 708. In other words, determining whether at least one of the likelihood estimates exceeds the threshold indicates convergence to a single likelihood estimate corresponding to a single subject, as shown in block 714. For example, device 104 or a component thereof (e.g., Figure 8 CGR content recommendation module 856 or Figure 5 The likelihood estimator 530 in the device 104 determines that at least one likelihood estimate value of at least one corresponding subject in the first group of subjects includes a first likelihood estimate value of the first subject and a second likelihood estimate value of the second subject. In response to determining that the multiple likelihood estimates together exceed the threshold, the device 104 updates the likelihood estimate value of each of the first group of subjects based on the updated user context and at least one of the updated first posture information, including generating an updated first likelihood estimate value of the first subject and an updated second likelihood estimate value of the second subject. The device 104 further selects between the first subject and the second subject based on the updated first likelihood estimate value and the updated second likelihood estimate value. Figure 3 As shown, frame 310 and vase 330 are tied during the second and third iterations. Using the updated likelihood estimates during the fourth iteration, the likelihood estimates converge to a single likelihood estimate corresponding to frame 310.

[0087] Method 700 continues at block 716, where the electronic device generates, in response to determining that at least one likelihood estimate exceeds a confidence threshold, a recommended content or action associated with at least one corresponding subject using at least one classifier associated with at least one corresponding subject and the user context. In some embodiments, device 104 or a component thereof (e.g., Figure 8 The CGR content rendering module 858 or the likelihood estimator 530 in the example of FIG. 5 ) allocates computationally intensive tasks, such as fine matching with the second computing device, as shown in block 718. In such embodiments, as described above with reference to Figure 5 As explained, the device 104 (e.g., Figure 5Device 104 then remotely sends the compressed vector to a second device to generate recommendation weights for a classifier associated with the first set of subjects. After performing fine-grained matching at the second device, for example, through machine learning across users, device 104 receives the recommendation weights from the second device for use in generating recommended content or actions. In some embodiments, to further facilitate recommendation generation, as shown in box 720, device 104 stores the first set of subjects and associated weights in multiple cascaded caches (e.g., Figure 5 In such an embodiment, the objects are stored in the cascaded caches 550-1, 550-2, 550-3, ... 550-N). In such an embodiment, the objects are stored in the cascaded caches in weight order.

[0088] In some embodiments, method 700 continues at box 722, where the electronic device predicts different subjects that exceed the confidence threshold based on the updated user context and at least one of the updated first gesture information, and generates a set of recommended content or actions associated with the different subjects. For example, if the first gesture and the second gesture indicate that the focus is moving to the right within the field of view, the likelihood estimator predicts the next subject on the right side of the field of view based on the user context to provide recommended content. For example, if Figure 4A As shown, initially, the focus associated with the first pose is on the frame at the center of the field of view. Continuing with this example, Figure 4B As shown, as the focus associated with the second gesture moves to the right, based on the user context of the meeting in the calendar, the user is more likely to want to view the clock on the right wall, even though gaze area 222 is still within the threshold distance of clock 240. In contrast, since the change from the first gesture to the second gesture indicates that the gaze area is shifting to the right, the user is less likely to want to obtain information about dog 236 on the left side. Based on the updated gesture information, device 104 predicts that the user is most likely interested in event calendar 242 associated with clock 240. However, as Figure 4C As shown, based on updated user context information, such as that the upcoming veterinarian visit is for dog 236 , device 104 predictively generates and provides more information about dog 236 and veterinarian appointment information (eg, recommended content 244 ).

[0089] Figure 8 is a block diagram of a computing device 800 according to some embodiments. In some embodiments, the computing device 800 corresponds to Figure 1104, and performs one or more of the above functions. Although some specific features are shown, those skilled in the art will recognize from the present disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the embodiments disclosed herein. To achieve this goal, as a non-limiting example, in some embodiments, the computing device 800 includes one or more processing units (CPUs) 802 (e.g., processors), one or more input / output (I / O) interfaces 803 (e.g., network interfaces, input devices, output devices, and / or sensor interfaces), memory 810, programming interfaces 805, and one or more communication buses 804 for interconnecting these components and various other components.

[0090] In some embodiments, one or more communication buses 804 include circuits for interconnecting and controlling communication between system components. Memory 810 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and in some embodiments, includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 810 optionally includes one or more storage devices remotely located from one or more CPUs 802. Memory 810 includes a non-transitory computer-readable storage medium. In addition, in some embodiments, memory 810 or a non-transitory computer-readable storage medium of memory 810 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 820, an image capture control module 850, an image processing module 852, a subject recognition module 854, a CGR content recommendation module 856, and a CGR content rendering module 858. In some embodiments, one or more instructions are included in a combination of logical and non-transitory memory. The operating system 820 includes processes for handling various basic system services and for performing hardware-related tasks.

[0091] In some embodiments, the image capture control module 850 is configured to control the functions of an image sensor or camera assembly to capture an image or obtain image data. To this end, the image capture control module 850 includes a set of instructions 851a and heuristics and metadata 851b.

[0092] In some embodiments, the image processing module 852 is configured to pre-process the raw image data from the image sensor or camera assembly (e.g., convert the RAW image data to RGB or YCbCr image data and derive pose information, etc.). To this end, the image processing module 852 includes a set of instructions 853a and heuristics and metadata 853b.

[0093] In some embodiments, the subject recognition module 854 is configured to recognize subjects from the image data. To this end, the subject recognition module 854 includes a set of instructions 855a and heuristics and metadata 855b.

[0094] In some embodiments, the CGR content recommendation module 856 is configured to recommend CGR content items associated with the identified subject. To this end, the CGR content recommendation module 856 includes a set of instructions 857a and heuristics and metadata 857b.

[0095] In some embodiments, the CGR content rendering module 858 is configured to synthesize and render CGR content items in a field of view close to the identified subject. To this end, the CGR content rendering module 858 includes a set of instructions 859a and heuristics and metadata 859b.

[0096] Although the image capture control module 850, the image processing module 852, the subject recognition module 854, the CGR content recommendation module 856, and the CGR content rendering module 858 are shown as residing on a single computing device, it should be understood that in other embodiments, any combination of the image capture control module 850, the image processing module 852, the subject recognition module 854, the CGR content recommendation module 856, and the CGR content rendering module 858 may reside in separate computing devices in various embodiments. For example, in some embodiments, each of the image capture control module 850, the image processing module 852, the subject recognition module 854, the CGR content recommendation module 856, and the CGR content rendering module 858 may reside on a separate computing device or in the cloud.

[0097] also, Figure 8 It is more of a functional description of various features present in a particular implementation, as opposed to a block diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 8 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are allocated among them will vary from implementation to implementation and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0098] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0099] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0100] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0101] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method comprising: At a device including non-transitory memory and one or more processors: detecting a first group of subjects having a corresponding first group of subject types within a field of view of the device; determining a value indicative of user interest for each subject in the first set of subjects, including determining a first value for a first subject in the first set of subjects having a first subject type in the corresponding first set of subject types; and In response to determining that the first value exceeds a threshold: causing a first classifier associated with the first subject type to be applied to the first subject to identify the identity of the first subject of the first subject type; and First content is generated based on the identity of the first subject. The method of claim 1 , wherein determining the first value is based on a user context. The method of claim 2 , wherein determining the first value is based on a relationship between the user context and the first subject type. The method of claim 1 , wherein determining the first value is based on a gaze of the user. The method of claim 1 , wherein determining the first value is based on a percentage of the first field of view occupied by the first subject. The method of claim 1 , wherein determining the first value is based on a change in a posture of the device.

7. The method of claim 1 , wherein determining the value indicative of user interest in each subject in the first set of subjects comprises: generating a compressed vector representing the first group of objects; sending the compressed vector to a second device to generate the value indicative of user interest in each subject in the first set of subjects; and The value indicative of user interest in each subject in the first group of subjects is received from the second device.

8. The method of claim 7, wherein the value indicative of user interest in each subject in the first set of subjects is based on machine learning applied across multiple users.

9. The method according to claim 7, further comprising: The first set of subjects and the value indicating user interest in each subject in the first set of subjects are stored in a plurality of cascading caches, the plurality of cascading caches being ordered by the value indicating user interest in each subject in the first set of subjects.

10. The method of claim 1, wherein determining that the first value exceeds the threshold comprises: The first value is compared to values ​​indicative of user interest in other subjects in the first group of subjects. The method according to claim 1 , wherein generating the first content comprises generating content matching the identity of the first subject. 12 . The method of claim 1 , wherein generating the first content comprises acquiring information about a subject having the first subject type and the identity.

13. The method according to claim 1, further comprising: causing a second classifier associated with a second subject type to be applied to a second subject in the first group of subjects to identify the identity of the second subject of the second subject type; as well as Second content is generated based on the identity of the second subject.

14. The method according to claim 13, further comprising: The first classifier associated with the first subject type and the second classifier associated with the second subject type are stored in the non-transitory memory.

15. The method according to claim 1, further comprising: The first content associated with the first subject is displayed.

16. An electronic device, comprising: non-transitory memory; and One or more processors configured to: detecting a first group of subjects having a corresponding first group of subject types within a field of view of the device; determining a value indicative of user interest for each subject in the first set of subjects, including determining a first value for a first subject in the first set of subjects having a first subject type in the corresponding first set of subject types; and In response to determining that the first value exceeds a threshold: causing a first classifier associated with the first subject type to be applied to the first subject to identify the identity of the first subject of the first subject type; and First content is generated based on the identity of the first subject.

17. The apparatus of claim 16, wherein the one or more processors are configured to determine the first value based on a user context.

18. The apparatus of claim 16, wherein the one or more processors are configured to determine the value indicating user interest in each subject in the first set of subjects by: generating a compressed vector representing the first group of objects; sending the compressed vector to a second device to generate the value indicative of user interest in each subject in the first set of subjects; and The value indicative of user interest in each subject in the first group of subjects is received from the second device.

19. The apparatus of claim 16, wherein the one or more processors are further configured to: causing a second classifier associated with a second subject type to be applied to a second subject in the first group of subjects to identify the identity of the second subject of the second subject type; and Second content is generated based on the identity of the second subject.

20. A non-transitory computer-readable medium having instructions encoded thereon, the instructions, when executed by an electronic device including a processor, causing the electronic device to: detecting a first group of subjects having a corresponding first group of subject types within a field of view of the device; determining a value indicative of user interest for each subject in the first set of subjects, including determining a first value for a first subject in the first set of subjects having a first subject type in the corresponding first set of subject types; and In response to determining that the first value exceeds a threshold: causing a first classifier associated with the first subject type to be applied to the first subject to identify the identity of the first subject of the first subject type; and First content is generated based on the identity of the first subject.

Images