Augmented reality system

Abstract

An AR system comprising a user interface, one or more sensors arranged to generate sensor data representing a portion of an environment in which a user of the AR system is located, and a memory. The memory is arranged to store object association data associating the user with one or more objects in the environment, and object position data indicating a respective position of each of the one or more objects. The AR system is arranged to determine a position of the user, determine an updated position of one of the one or more objects in dependence on the generated sensor data and the determined position of the user, update the stored object position data to indicate the determined updated position of the one of the one or more objects, and output information in dependence on the updated position of the one of the one or more objects via the user interface.

Description

Technical Field

[0001] This invention relates to augmented reality (AR) systems. This invention has a specific, but not exclusive, relevance to the use of AR systems for tracking a user's property. Background Technology

[0002] For example, losing property by leaving it behind when leaving a public place can cause significant inconvenience and can be very costly if the property cannot be found or recovered. This problem can be particularly common for people with memory problems, such as those caused by dementia, and / or for small items that may not be easily noticed when they are missing. For example, people often leave home with a smartphone, a pair of headphones, a set of keys, a wallet, and / or a handbag.

[0003] Augmented reality (AR) devices, such as AR headsets, that provide visual information to enhance the user’s experience of the environment are becoming smaller, lighter, and have shape factors that allow for extended and / or everyday use. Summary of the Invention

[0004] According to a first aspect, an augmented reality (AR) system is provided. The AR system includes a user interface, one or more sensors arranged to generate sensor data representing a portion of an environment in which a user of the AR system is located, and a memory. The memory is arranged to store: object association data associating the user with one or more objects in the environment, and object location data indicating the corresponding location of each of the one or more objects. The AR system is arranged to: determine the user's location; determine an updated location of one of the one or more objects based on the generated sensor data and the user's determined location; update the stored object location data to indicate the determined updated location of said one of the one or more objects; and output information via the user interface based on the updated location of said one of the one or more objects.

[0005] In the example, the sensor data includes image data, and determining the updated location of said one of one or more objects includes processing the image data to detect said one of one or more objects.

[0006] In the example, determining the updated location of said one of one or more objects includes determining whether the user currently holds said one of one or more objects, and the updated object location data indicates whether the user currently holds said one of one or more objects.

[0007] In the example, the AR system is configured to detect or anticipate separation of one of one or more objects from the user based on updated location data, generate an alarm based on the detected or anticipated separation of the one of one or more objects from the user, and output the alarm via a user interface. The alarm may include information indicating the updated location of the one of one or more objects.

[0008] In the example, the memory is further arranged to store privileged location data indicating one or more privileged locations, and to generate an alarm based on whether the updated location of an object corresponds to any one of the one or more privileged locations.

[0009] In the example, the system is configured to use the user's determined location to detect or anticipate the user leaving the privileged location, and to generate an alarm based on the detection or anticipation of the user leaving the privileged location.

[0010] In the example, the AR system is configured to detect pickup of one of one or more objects by a person other than the user based on generated sensor data, and to detect or anticipate separation of the one of one or more objects from the user based on the detected pickup of the one of one or more objects by a person other than the user. The AR system may be configured to capture an image of the person other than the user when it detects that the person has picked up the one of one or more objects.

[0011] In the example, the information output via the user interface is in response to a request from the user. At least one of speech recognition and gesture recognition can be used to derive the request.

[0012] In the example, the AR system is configured to update stored object association data based on additional sensor data generated by one or more sensors to associate a user with additional objects in the environment. The additional sensor data may include additional image data, and updating the stored object association data may involve processing the additional image data to learn the appearance of the additional objects. The updating of the object association data may depend on input from the user, for example, derived using at least one of speech recognition and gesture recognition.

[0013] In the example, the AR system is configured to use simultaneous localization and mapping SLAM to determine the user's location.

[0014] In the example, the user interface includes one or more displays, and the AR system is arranged to visually output information on one or more displays.

[0015] In the example, one of the one or more objects is a device arranged to transmit wireless signals, and the AR system is arranged to use the transmitted wireless signals to identify the one of the one or more objects.

[0016] According to a second aspect, a computer-implemented method is provided. The method includes: storing object association data that associates one or more objects in an environment with a user of an AR system; receiving sensor data representing a portion of the environment in which the user is located; determining the user's position; determining the position of one of the one or more objects associated with the user based on the received sensor data and the user's determined position; storing object position data indicating the determined position of said one of the one or more objects; and outputting information via a user interface of the AR system based on the determined position of said one of the one or more objects.

[0017] Other features and advantages of the invention will become apparent from the following description of preferred embodiments of the invention, given by way of example only with reference to the accompanying drawings. Attached Figure Description

[0018] Figure 1 This is a schematic block diagram illustrating an augmented reality (AR) system based on an example;

[0019] Figure 2 An example of an AR system including a pair of smart glasses is shown;

[0020] Figure 3 This is a flowchart illustrating a computer-implemented method for generating information for output via a user interface of an AR system, based on an example;

[0021] Figure 4A and Figure 4B It shows the result of Figure 2 An example of information output by an AR system;

[0022] Figure 5 This is a flowchart illustrating a computer implementation of a method for generating an alarm for output via a user interface of an AR system, based on an example.

[0023] Figures 6A to 6C It shows the result of Figure 2 An example of an alarm output by an AR system;

[0024] Figure 7 This is a flowchart illustrating a further computer-implemented method for generating an alarm for output via a user interface of an AR system, based on an example; and

[0025] Figure 8This is a flowchart illustrating a computer-implemented method for generating data that associates a user of an AR device with objects in the environment, based on an example. Detailed Implementation

[0026] Figure 1 An example of an AR system 100 is shown. The AR system 100 can be embodied as a single AR device, such as a headset, a pair of smart glasses, or any other suitable type of wearable device. Alternatively, the AR system 100 may include multiple devices connected via wired or wireless means. (See reference...) Figure 2 This specific implementation will be described in more detail.

[0027] AR system 100 includes one or more sensors 102 arranged to generate sensor data representing a portion of the environment in which a user of AR system 100 is located. Sensor 102 may be a component of a single device such as an AR headset, or alternatively, a component of multiple connected devices. Sensor 102 includes one or more cameras for generating image data representing a portion of the environment falling within the field of view of the one or more cameras. The field of view may be defined in a vertical and / or horizontal direction, depending on the number and location of the cameras. For example, the cameras may be arranged to face substantially the same direction as the head of a user wearing an AR headset, in which case the field of view of the one or more cameras may include all or part of the user's field of view. Alternatively, the field of view may include a wider area, such as completely surrounding the user. The cameras may include stereo cameras, from which the AR system can use stereo matching to derive depth information indicating the distance to objects in the environment. Alternatively or otherwise, sensor 102 may include a depth sensor for generating depth information, such as an infrared camera, a sonar transceiver, and / or a light detection and ranging (LIDAR) system. AR system 100 can be configured to combine image data and associated depth information to generate a three-dimensional representation of a part of the environment, such as in RGB-D format, and / or as a point cloud or volumetric representation.

[0028] Sensor 102 includes a position sensor for determining the location and / or orientation (collectively referred to as position or posture) of the user of AR system 100. The position sensor may include a Global Positioning System (GPS) module, one or more accelerometers, and / or a Hall effect magnetometer (electronic compass) for determining orientation. Alternatively or additionally, AR system 100 may be able to determine or refine the user's estimated location by analyzing image data and / or depth information using Simultaneous Localization and Mapping (SLAM) techniques.

[0029] AR system 100 includes a user interface 104 through which a user can interact with AR system 100. User interface 104 includes input devices and output devices, which may be components of a single AR device or alternatively components of multiple connected devices. Output devices are arranged to provide information to the user and include one or more displays for providing visual information to the user to enhance the user's environmental experience. The one or more displays may include opaque displays arranged to generate and display image data corresponding to a representation of a portion of the environment generated using one or more cameras and / or depth sensors, wherein additional information or virtual objects overlap or otherwise combine with the generated representation of the environment. Alternatively or concurrently, the one or more displays may include transparent displays through which the user can directly observe the environment and project information or virtual objects, for example, using waveguide or laser scanning display technology.

[0030] The output device may include one or more speakers, such as those mounted in a handset or headphones, allowing the AR system 100 to output information to the user in the form of audio. The audio may include, for example, synthesized or pre-recorded speech, blew sounds, beeps, clicks, music, or any other sound suitable for conveying information to the user.

[0031] The output device may also include a haptic output device arranged to generate forces to cause some or all of the movement of the AR system 100, including, for example, vibration, click, or other movements detectable by the user's touch. In one example, the AR headset may send signals to another device, such as a smartwatch, fitness tracker, bracelet, or other wearable device or smartphone, enabling that other device to provide haptic output to the user.

[0032] The input device of the user interface 104 is arranged to receive information from the user of the AR system 100. The input device may include one or more microphones for capturing speech or other sounds emitted by the user. For example, the input device may include a microphone array from which the AR system 100 can determine the direction to an audio source, thereby allowing the AR system 100 to distinguish the sound emitted by the user from other sounds in the environment. The AR system 100 may also be arranged to perform speech recognition and respond to verbal commands from the user.

[0033] Input devices may include one or more eye-tracking sensors arranged to track the orientation and / or movement of a user's eyes. Eye-tracking sensors may, for example, be optical eye-tracking sensors capable of tracking eye orientation by analyzing images of the eyes generated by a camera facing the eyes. The eye-tracking sensors may generate eye-tracking data from which the AR system 100 can determine which part of the environment or which object in the environment the user is currently looking at. The eye-tracking sensors may also be used to determine when the user blinks or closes his or her eyes, which the AR system 100 may use as an input signal.

[0034] The input device may also include a button or touch input device. For example, AR system 100 may include one or more scroll wheels, touch-sensitive areas, or touchpads. As described above, the input device may be part of the AR device housing sensor 102, or it may be part of a separate remote device.

[0035] In some examples, one or more cameras of the AR system 100 can also be used as user input devices, for example, to facilitate gesture recognition. Additionally, accelerometers and / or electronic compasses can be used to determine when a user nods or shakes their head.

[0036] AR system 100 includes memory 108 and processing circuitry 110. Memory 108 and processing circuitry 110 may be part of an AR device housing sensor 102. Alternatively, some of memory 108 and processing circuitry 110 may be part of one or more separate devices, such as dedicated computing devices, smartphones, tablets or laptops, desktop computers, servers, or one or more devices in a networked system. In this example, some data storage and processing tasks occur locally at the AR device, while others occur remotely. This allows the data storage and processing performed by the AR device to be kept to a necessary minimum, thereby allowing the AR device to have a practical and attractive size, weight, and form factor for long-term and / or everyday use.

[0037] The memory circuitry 108 includes, for example, one or more solid-state drives (SSDs) of non-volatile memory, and non-volatile and volatile random access memory (RAM), such as static random access memory (SRAM) and dynamic random access memory (DRAM). Other types of memory may be included, such as removable memory devices, synchronous DRAM, etc.

[0038] Processing circuitry 110 may include various processing units, including a central processing unit (CPU), a graphics processing unit (GPU), and / or a dedicated neural processing unit (NPU) for efficiently performing neural network operations. For the purposes of this invention, the neural network may be used for certain tasks, including object detection and SLAM, as will be described in more detail below. Processing circuitry 110 may also include other expert processing units, such as application-specific integrated circuits (ASICs), digital signal processors (DSPs), or field-programmable gate arrays (FPGAs).

[0039] Memory 108 holds machine-readable instructions in the form of program code that, when executed by processing circuitry 110, cause AR system 100 to perform the methods described below. Memory 108 is also arranged to store additional data for performing the methods. This additional data in this example includes sensor data generated by one or more sensors 102, object association data that associates the user of AR system 100 with one or more physical objects in the environment, and object location data indicating the corresponding location of each of the one or more objects.

[0040] Figure 2 An example of an AR system 200 is shown, which includes a pair of smart glasses 202 and an associated application on a smartphone 203. In this example, the smart glasses 202 and the smartphone 203 pair during a pairing process, but in other examples, the smart glasses 202 may pair with another type of device such as a smartwatch or tablet, or alternatively, may function without pairing with any other device.

[0041] The smart glasses 202 include a central frame portion 204 and two folding arms 206a, 206b, wherein the central frame portion 204 serves as a support for the two lenses 208a, 208b. The central frame portion 204 and the arms 206a, 206b house various sensors and user interface components, as described below. In this example, the lenses 208a, 208b are neutral; however, in other examples, the lenses may be corrective lenses matched to a specific user's prescription, and / or may be tinted, as in the case of smart sunglasses. Each of the lenses 208a, 208b is a transparent display, on which corresponding projection components 210a, 210b are arranged to display information to the user.

[0042] The central frame portion 204 houses two forward-facing cameras 212a and 212b, with the combined field of view roughly corresponding to the user's field of view. The AR system 200 is arranged to use stereo matching to analyze image data generated by the cameras 212a and 212b to determine depth information. The central frame portion 204 also houses a microphone array 214 for receiving voice input from the user, and optical eye-tracking sensors 216a and 216b for tracking the orientation and movement of the user's right and left eyes, respectively. The arms 206a and 206b house dedicated power supplies, processing circuitry, and memory circuitry, as well as a Global Positioning System (GPS) receiver, an electronic compass, an accelerometer, and a communication module including an antenna for wireless communication with the smartphone 203 running the associated application. It should be noted that while the smart glasses 202 in this example includes the necessary components for the smart glasses 202 to function independently of the smartphone 203, the smartphone 203 also includes some equivalent components of the smart glasses 202, such as the GPS receiver and the accelerometer. Where appropriate, AR system 200 may use components of smartphone 203 to replace equivalent components of smart glasses 202, for example, to save battery power of smart glasses 202.

[0043] AR system 200 is configured to determine the user's location (i.e., position and orientation) using the vehicle-mounted GPS receiver and electronic compass of smart glasses 202 and / or by using SLAM processing of image data from cameras 212a, 212b. AR system 200 may be configured to continuously monitor the user's location, or alternatively to determine the user's location only when certain events are detected (e.g., when the accelerometer detects movement of the user).

[0044] The smart glasses 202 can be configured according to user preferences, such as using an application on the smartphone 203 or directly using the input devices of the smart glasses 202. For example, the user can use the application to select the type of information displayed on lenses 208a, 208b, and whether the smart glasses 202 continuously monitors the user's location as described above. The application has associated storage on the smartphone 203, which, in addition to the memory circuitry of the smart glasses 202, can also be used to store data for use by the AR system 200. The AR system 200 can also utilize the processing power of the smartphone 203 to perform certain resource-intensive processing tasks, such as SLAM. Sharing the storage and processing requirements of the AR system 200 between the smart glasses 202 and the smartphone 203 allows the size, weight, and shape factors of the smart glasses 202 to resemble a pair of regular glasses, enabling the user to wear the smart glasses 202 comfortably for extended periods and daily.

[0045] Figure 3An example of a method performed by an AR system 200 according to the present invention is shown. It should be understood that although the method is described with reference to AR system 200, any suitable embodiment of AR system 100 may perform the same method without departing from the scope of the invention.

[0046] At position 302, AR system 200 stores object association data associated with one or more objects in the environment. AR system 200 can store association data for multiple users. In this example, an application on smartphone 203 can be used to view the object association data. AR system 200 stores object association data for user John Doe, indicating that John Doe is associated with three objects—a set of keys, a wallet, and smartphone 203. In this example, the application displays a table by user's name, with rows corresponding to different objects associated with the user. Columns titled "O" include icons representing the objects. Columns titled "P" indicate whether the user currently holds an object. Columns titled "L" indicate the location of the object. Reference Figure 8 Methods for generating, storing, and updating object-related data are described.

[0047] At 304, AR system 200 receives sensor data representing a portion of the environment in which the user of AR system 200 is located. In this example, the sensor data includes image data generated by forward-facing cameras 212a and 212b. Cameras 212a and 212b are arranged to continuously generate image data frames for analysis by AR system 200. The image data frames are generated at a rate high enough to capture events occurring in the environment but low enough to allow AR system 200 to analyze the image data in real time. In other examples, sensor data may include other types of data, such as depth information.

[0048] At 306, AR system 200 determines the location of the user. Based on the information available to AR system 200, the determined location can be a global location or a local location relative to a local coordinate system of the user's room or other nearby area. In this example, AR system 200 is configured to determine the user's location and orientation using the onboard GPS receiver, accelerometer, and / or electronic compass of smart glasses 202 and / or by processing image data from cameras 212a, 212b. For example, AR system 200 can use the GPS receiver and electronic compass to determine the user's approximate location and orientation, and then, if the user is in a suitable location (e.g., if the user is indoors), use SLAM to determine the precise location and orientation. Alternatively, if AR system 200 cannot use the GPS receiver to determine the user's global location, AR system 200 can use SLAM to determine the local location. In addition or alternatively, AR system 200 can use computer vision techniques such as scene recognition to determine the user's location. For example, AR system 200 can use scene recognition to determine whether the user is in a bar or restaurant, or in the user's home.

[0049] At 308, AR system 200 determines the position of an object among those indicated as associated with the user, based on sensor data received at 306 and the user's position determined at 308. In this example, AR system 200 uses object detection to process image data generated by cameras 212a and 212b. Upon detecting an object among those associated with the user, AR system 200 determines the position of the detected object relative to the user. In this example, depth information derived using stereo matching is used to determine the position of the detected object relative to the user in three dimensions. AR system 200 then determines the object's position based on the user's determined position. The object's position can be a global position, a local position relative to a local coordinate system, or a combination of both. For example, if the user's position determined at 306 is a global position, AR system 200 can determine the object's global position. If the user's position determined at 306 is a local position, AR system 200 can determine the object's local position.

[0050] In addition to determining the location of the detected object, AR system 200 also determines whether the user is currently holding the detected object, such as holding the object, carrying the object in a pocket, or wearing the object if it is clothing. To this end, AR system 200 is configured to recognize when the user picks up or puts down the object. Because cameras 212a and 212b are forward-facing, their field of view is approximately equal to the user's field of view, so when the user picks up or puts down the object, the object will most likely be within the field of view of cameras 212a and 212b. In this example, to recognize when the object is picked up or put down, a machine learning classifier based on a convolutional neural network is trained to recognize when the user holds the object in their hand. If, in the image sequence captured by cameras 212a and 212b, it is determined that the object remains in the user's hand while leaving the field of view of cameras 212a and 212b, then AR system 200 will determine that the object has been picked up by the user, and the user is therefore holding the object. In contrast, if in an image sequence it is first determined that an object is in the user's hand and then it is subsequently determined that the object is no longer in the user's hand but is still in the field of view, then AR system 200 can determine that the object has been put down or placed in the environment, and therefore the user no longer holds the object. It should be understood that other methods can be used to identify when a user picks up or puts down an object. More generally, if AR system 200 detects an object in the environment and the object is not held by the user, then AR system 200 will determine that the user is not currently holding the detected object. At 310, AR system 200 stores object location data indicating the determined object's location. The object location data may, for example, include precise global or local coordinates of the determined location of a given object, including latitude, longitude, and altitude. Other representations are also possible without departing from the scope of the invention. For example, it has been shown that if the surface of the Earth is divided into a grid of three-meter squares, each of the three-meter squares can be uniquely identified using an arrangement of three words in the English language.

[0051] In addition to the coordinate representation of the determined object location, AR system 200 can also determine names or other identifiers, such as the updated location's postal code, by interacting with local or cloud-based mapping software and / or by recognizing user-specified locations such as "home," "work," etc. By using the user-defined location and analyzing image data received from cameras 212a, 212b, AR system 200 may be able to determine very specific identifiers of the object location, such as "on your work desk" or "on the bar counter at Eagle Bar." In this example, the object location data also indicates whether the user is currently holding the object.

[0052] In another example, the AR system can store object location data in a hierarchical manner. For instance, the AR system can identify when a first object is placed inside a second object, such as a bag, thus establishing a correlation between the positions of the first and second objects. The object location data will then indicate that the position of the first object is the same as the position of the second object until the AR system 200 determines that the positions of the two objects are no longer correlated. Similarly, the object location data can indicate that a first object, such as a credit card, is located inside a second object, such as a handbag, which is located inside a third object, such as a handbag. The AR system 200 will establish a correlation between the positions of the three objects such that the object location data indicates that the positions of all three objects are the same as the position of the handbag, until the AR system 200 determines that the positions of the three objects are no longer correlated.

[0053] At 312, AR system 200 outputs information based on stored object location data. This information may, for example, indicate the most recently determined location of an object associated with the user. In one example, AR system 200 outputs information in response to a request from the user. This request may include a verbal request received via microphone array 214, in which case AR system 200 may use speech recognition and natural language processing to recognize the request and / or determine its content. The verbal request may, for example, include the user asking, “Where are my keys?” Alternatively, the request may include a gesture, in which case AR system 200 may recognize the request by analyzing image data received from cameras 212a, 212b. The request may be received via any other suitable input method.

[0054] AR system 200 can output information to the user through any user interface of AR system 200, such as by displaying information on lenses 208a and 208b of smart glasses 202. Figure 4A An example is shown in which information indicating the location of an object associated with the user is displayed. The displayed information includes an icon 402 representing the object (a set of keys), an arrow 404 indicating the direction to the most recently determined location of the object, and text indicating the distance to the most recently determined location of the object. Figure 4B Another example is shown, which displays information indicating the location of an object associated with the user. The displayed information includes an icon 406 representing the object (wallet) and a text description of the object's most recently determined location. In another example, this information may include images captured by cameras 212a, 212b, showing the location where the AR system 200 last detected the object.

[0055] As an alternative to displaying information on lenses 208a and 208b, the AR system 200 can output information via an application on a smartphone 203. Figure 2In the example, the application displays a table indicating the location data of several objects associated with the user. The column titled "P" indicates whether the user, John Doe, currently possesses each object ("Y" indicates the user currently possesses the object, and "N" indicates the user currently does not possess the object). In this case, John Doe does not currently possess the set of keys or wallet, but currently possesses smartphone 203. The column titled "L" indicates the location of each object. In this example, the table does not display the location of smartphone 203 because it has already been determined that John Doe currently possesses smartphone 203.

[0056] In the example above, information is visually displayed to the user of AR system 200. In other examples, the information may be delivered to the user via any other suitable method, such as audio including synthesized or pre-recorded speech.

[0057] Figure 5 An alternative method is shown, performed by AR system 200 (or any other suitable implementation of AR system 100). Figure 5 Operations 502-510 and Figure 3 Operations 302-310 are the same. At 512, AR system 200 detects or anticipates separation of the user from one of the objects associated with the user. For example, separation can be detected if the object is in the field of view of cameras 218a, 218b, and the object or the user moves such that the object is no longer in the field of view. Separation can be anticipated, for example, if the angular separation of the object from the axes of cameras 212a, 212b exceeds a predetermined threshold, indicating that the user has moved away from the object.

[0058] If a user places an object in a specific location, the AR system 200 can anticipate the object's separation from the user. For example, if the AR system 200 detects that a user has placed a set of keys on a surface in a public place, the AR system 200 can anticipate the keys' separation from the user. The AR system 200 can also use eye-tracking sensors 216a and 216b to determine whether the user is looking directly at the object when placing it. If the user is not looking directly at the object, the AR system 200 can determine that the user is not focused on the object and therefore anticipates the object's separation from the user.

[0059] When the object is outside the field of view of cameras 212a and 212b, AR system 200 can further predict or detect separation between the object and the user. For example, AR system 200 can detect that the user is moving away from the object from the user's position determined at 506, and thus detect separation between the user and the object. For example, separation can be detected if the distance between the user and the object increases to exceed a predetermined threshold. Alternatively, AR system 200 can detect separation between the user and the object by determining that the user is leaving the building where the object is located without holding the object. For example, AR system 200 can recognize that the user has placed a set of keys on a table in a restaurant and store object location data indicating the location of the keys. Later, AR system 200 can detect separation between the user and the set of keys by determining that the user is leaving the restaurant without holding the set of keys.

[0060] If AR system 200 determines that an object has moved within the field of view of cameras 212a and 212b, separation can also be detected or anticipated. For example, the object may have moved if someone other than the user has picked it up (which could indicate that the object is being stolen), or if the object has fallen from a surface. AR system 200 can also detect the position or movement of an object when it is not within the field of view of cameras 212a and 212b. For example, AR system 200 can detect the movement of a connected device (such as smartphone 203) based on wireless signals (such as Bluetooth or Wi-Fi signals) transmitted by the connected device.

[0061] At 514, AR system 200 generates an alarm based on the detected or anticipated separation of an object from the user, and at 516, outputs the alarm via the user interface. In this example, the alarm is a visual alarm projected onto lenses 208a, 208b of smart glasses 202. In other examples, the alarm may optionally or additionally have an audio component, such as pre-recorded or synthesized speech or any other sound. The alarm may identify the object and / or may include information indicating the location of the object as indicated by object location data. In some examples, different levels of alarms may be generated based on different events. For example, in response to the user moving away from the object so that the object is no longer in the field of view of cameras 212a, 212b, AR system 200 may generate a first alarm when separation of the user from the object is anticipated. AR system 200 may then generate a second alarm upon detecting separation of the user from the object, such as if the user moves away from the object or if the user leaves the building where the object is located. While the first alert can be a relatively inconspicuous visual alert, such as an arrow displayed around the periphery of lenses 208a and 208b, the second alert can be more intrusive, such as being centered on lenses 208a and 208b, and / or including an audio component. If the AR system 200 detects that an object has been stolen, the AR system 200 can generate a high-priority alert, such as one that includes an audio component. Figures 6A to 6C An example of generating an alarm for a user of AR system 200 is shown. Figure 6A In this example, AR system 200 has identified the user's smartphone 203 on table 604. A border 602 is displayed on lens 208b to indicate to the user that AR system 200 has identified the smartphone 203. Stored object location data is updated to indicate the position of the smartphone 203 on table 604. Figure 6B In the image, the user has begun moving in the direction indicated by arrow A. Figure 6C In this example, the user continues to move in the direction indicated by arrow A, causing smartphone 203 to fall out of the field of view of cameras 212a and 212b. When the position of smartphone 203, as indicated by the object location data, moves from inside the field of view of cameras 212a and 212b to outside the field of view, the AR system 200 detects the separation of the user from smartphone 203. The AR system 200 then generates and displays an alert on lens 208b in the form of arrow 606, pointing to the position of smartphone 203 indicated by the object location data.

[0062] Figure 7 An alternative method is shown, performed by AR system 200 (or any other suitable implementation of AR system 100). Figure 7 Operations 702-712 and Figure 5The operations are the same as 502-512, except that at 702, AR system 200 also stores privileged location data indicating one or more privileged locations. Privileged locations may include, for example, a user's home and / or a user's workplace. In this example, the user can enter a privileged location by manually entering the location details or by indicating that the user is currently located at a privileged location via an application on smartphone 203. In other examples, the AR system may automatically identify a location as the user's home, for example, by analyzing the user's location over a long period of time. In this example, the privileged location data indicates a set of precise coordinates for each privileged location; however, in other examples, the privileged location data may include additional information, such as the layout of the home or other building generated using SLAM.

[0063] At 712, after detecting or anticipating separation between objects indicated in the user-object association data, the AR system 200 determines at 714 whether the object location indicated by the object location data corresponds to any one of one or more privileged locations. In this example, if the object's location is less than a threshold distance from the coordinates indicated by the privileged location, the object's location is determined to correspond to that privileged location. In other examples, for instance, where the privileged location includes the layout of a home or other building, the AR system can accurately determine whether the object's location is within the building and therefore corresponds to a privileged location.

[0064] In this example, if AR system 200 determines that the object's location corresponds to a privileged location, AR system 200 takes no further action regarding the object's detected or anticipated separation from the user. Specifically, if the object remains at the privileged location, AR system 200 is prevented from alerting the user. For example, a user may choose to leave one or more objects at home; in this case, AR system 200 will not expect to alert the user when they leave home. Even if the user accidentally leaves an object at home, this may result in less inconvenience and / or cost compared to accidentally leaving an object elsewhere. If AR system 200 determines that the object is not located at a privileged location, the AR system generates and outputs an alarm at 716 and 718, as shown below. Figure 5 As described in sections 514 and 516.

[0065] Despite Figure 7 In the example, AR system 200 is configured to generate an alarm when it is determined that an object is not in a privileged location; however, AR system 200 can alternatively be configured to generate an alarm when it is determined that an object is in a privileged location. Furthermore, different privileged locations may apply to different objects. For example, if a user leaves home without an object belonging to a specified set of objects (e.g., a set of keys or a smartphone), the user may want to be notified, but if the user leaves home without an object not belonging to a specified set of objects, the user may not want to be notified.

[0066] Figure 8 An example of a method performed by AR system 200 (or any other suitable implementation of AR system 100) to store object association data that associates the user of AR system 200 with new objects not previously known to AR system 200. At 802, AR system 200 receives image data generated by cameras 212a, 212b. In this example, an application on smartphone 203 prompts the user to keep the new object in the field of view of cameras 212a, 212b in different orientations, causing cameras 212a, 212b to capture images of the object in various different orientations.

[0067] At 804, AR system 200 learns the appearance of objects. In this example, AR system 200 processes images of objects in different orientations to train a classifier using supervised learning, where the images of objects in different orientations are used as training data. AR system 200 thus learns to recognize objects from various angles and is therefore able to recognize objects when they appear in subsequent images. In this example, AR system 200 associates a unique identifier with an object, and AR system 200 is trained to determine the coordinates of a bounding box containing the object in addition to the unique identifier when an object is detected in an image. In this example, AR system 200 is pre-trained to recognize certain general categories of objects, such as "smartphone" or "key." This allows AR system 200 to efficiently detect objects of general categories, thus reducing the difficulty of training AR system 200 to learn the appearance of specific objects. This principle is called transfer learning.

[0068] At point 806, the AR system 200 has learned the appearance of the object, thereby storing object association data that indicates the association between the user and the object. In this example, the object association data includes a unique identifier for the object, an icon or image representing the object, and optionally, the name of the object.

[0069] exist Figure 8 In one example, AR system 200 learns the appearance of an object after prompting the user to keep it in the view of cameras 212a, 212b. In other examples, the AR system may automatically generate object association data, for example, where the AR system detects a specific object that frequently appears in the user's home or another privileged location. In other examples, the user may place a unique AR tag or marker on an object, training the AR system to detect that object. In yet another example, the AR system may learn other characteristics of the object, such as the identity of the connected device acquired during the wireless pairing process. The object association can then include the device's identity as an alternative to or supplement to the device's appearance. The AR system can then identify the device based on the wireless signals transmitted by the device.

[0070] The above embodiments should be understood as exemplary examples of the invention. Other embodiments of the invention are contemplated. For example, an AR system may be trained to recognize certain people other than the user of the AR system (e.g., members of the user's household or friends). The AR system may identify one of the identified people picking up an object and store object location data indicating that the object is held by that person. Furthermore, if the AR system identifies someone other than the user picking up an object, the AR system determines whether the user is one of the identified people. If the person is identified, the AR system does not take further action. If the person is not identified, the AR system issues an alarm to indicate that the person may be stealing an object. The AR system may also be configured to capture and store images of people who may be stealing objects for subsequent identification. In addition, or alternatively, alarm generation may depend on whether a person picks up an object in a privileged location. For example, if the AR system identifies another person in the user's home picking up an object, the AR system may not take further action. AR systems with a wide field of view, such as those completely surrounding the user, are particularly suitable for such applications.

[0071] It should be understood that any feature structure described with respect to any embodiment may be used alone or in combination with other described feature structures, and may also be used in combination with one or more feature structures of any other embodiment, or in combination with any feature structure of any other embodiment. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the invention as defined in the appended claims.

Claims

1. An augmented reality (AR) system, comprising: user interface; One or more sensors are arranged to generate sensor data, the sensor data including first image data representing a portion of the environment in which the user of the AR system is located at a first time and second image data representing a portion of the environment in which the user of the AR system is located at a second time; and The memory, which is arranged to store: Object association data that associates the user with multiple objects in the environment, including a first object and a second object, wherein the first object can be located within the second object; and Object location data indicating the corresponding position of each of the plurality of objects. The AR system is arranged as follows: By detecting the first object and the second object in the first image data, it is determined that the first object is located within the second object; Determine the user's location at the second time. The location of the second object is determined by detecting the second object in the second image data and based on the location determined by the user at the second time. Update the hierarchical object location data to indicate: The determined position of the second object; and The association between the first object and the second object; The stored object location data indicates that the location of the first object is the same as the location of the second object, until the AR system determines that the locations of the first object and the second object are no longer associated; The user interface outputs information indicating that the first object is located at a determined position of the second object based on the updated stored object location data.

2. The AR system according to claim 1, wherein: Determining the location of the second object includes determining whether the user currently holds the second object; and The object location data indicates whether the user currently owns the second object.

3. The AR system according to claim 1 or 2, wherein the AR system is arranged as follows: The separation of the second object from the user is detected or anticipated based on the updated stored location data; An alert is generated based on the detected or anticipated separation of the second object from the user; and The alarm is output via the user interface.

4. The AR system according to claim 3, wherein: The memory is further arranged to store privileged location data indicating one or more privileged locations; and The alarm is generated based on whether the determined location of the second object corresponds to any one of the one or more privileged locations.

5. The AR system of claim 4, wherein the AR system is configured to use the user's determined location to detect or anticipate the user leaving one of the one or more privileged locations. The generation of the alarm is based on the detection or expectation that the user leaves one of the one or more privileged locations.

6. The AR system of claim 3, wherein the AR system is configured to use the second image data to detect pickup of the second object by a person other than the user. The detection or expectation of the separation of the second object from the user is based on the picking of the second object by a person other than the user.

7. The AR system of claim 6, wherein the AR system is configured to capture an image of the person other than the user when the picking up of the second object is detected by the person other than the user.

8. A computer-implemented method, comprising: storage: Associating multiple objects in the environment with object data related to the user of the AR system, wherein the multiple objects include a first object and a second object, and the first object can be located within the second object; and Object location data indicating the corresponding position of each of the plurality of objects; Receive first image data representing a portion of the environment in which the user is located at a first moment; By processing the first image data to detect the first object and the second object, it is determined that the first object is located within the second object; Receive second image data representing a portion of the environment in which the user is located at a second time; Determine the user's location at the second time. The location of the second object is determined by detecting the second object in the second image data and based on the location determined by the user at the second time. Update the hierarchical object location data to indicate: The determined position of the second object; and The association between the first object and the second object; The stored object location data indicates that the location of the first object is the same as the location of the second object, until the AR system determines that the locations of the first object and the second object are no longer associated; as well as The AR system outputs information based on updated stored object location data via its user interface, indicating that the first object is located at a determined location of the second object.

9. A computer program product including machine-readable instructions, which, when executed by a computing system, cause the computing system to perform operations including: storage: Associating multiple objects in the environment with object data related to the user of the AR system, wherein the multiple objects include a first object and a second object, and the first object can be located within the second object; and Object location data indicating the corresponding position of each of the plurality of objects; Receive first image data representing a portion of the environment in which the user is located at a first moment; By processing the first image data to detect the first object and the second object, it is determined that the first object is located within the second object; Receive second image data representing a portion of the environment in which the user is located at a second time; Determine the user's location at the second time. The location of the second object is determined by detecting the second object in the second image data and based on the location determined by the user at the second time. Update the hierarchical object location data to indicate: The determined position of the second object; and The association between the first object and the second object; The stored object location data indicates that the location of the first object is the same as the location of the second object, until the AR system determines that the locations of the first object and the second object are no longer associated; as well as The AR system outputs information based on updated stored object location data via its user interface, indicating that the first object is located at a determined location of the second object.

Images