Content Output Management in Head-Mounted Wearable Devices

Abstract

A system and method are provided for selectively outputting content with a head-mounted wearable computing device. The system can determine a context associated with the operation of the head-mounted wearable computing device and, based on the context, selectively output content to the head-mounted wearable computing device or delay the output of the content. The content can be displayed on one or more designated portions of a display of the head-mounted wearable computing device to reduce user distraction and enhance situational awareness and situational safety during use of the head-mounted wearable computing device.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application is a continuation of U.S. Application No. 17 / 812,903, entitled "CONTENT OUTPUT MANAGEMENT IN A HEAD MOUNTED WEARABLE DEVICE", filed on July 15, 2022, and claims priority. The disclosure of this U.S. application is hereby incorporated by reference in its entirety into this specification.

[0002] This description generally relates to the management of content output in a head - mounted wearable device, and more particularly, to improving situational safety based on selective output of content in a head - mounted wearable device based on a situational context.

Background Art

[0003] Wearable devices can include devices worn on the head, such as glasses including smart glasses, headsets, earphones, etc., and devices worn on the wrist and / or hand, such as watches including smartwatches, smart bracelets, smart rings, etc., smart pendants, fitness trackers, cameras, body sensors, and other such devices. In some situations, for example, distraction due to interaction with the user's displayed content, incoming call notifications, etc. can compromise safety. In the case of smart glasses, not only can distraction be caused by the displayed content, but the user's view of the surroundings can also be blocked. For example, in some situations, such as professional interactions and / or social interactions, the user may wish to avoid these types of distractions so as to remain engaged in the interaction.

Summary of the Invention

[0004] In a first general aspect, a method executed by a computer includes determining a context associated with the operation of a head-mounted wearable device, determining a content output state of the head-mounted wearable device based on the context and one or more situational safety factors associated with the context, and outputting content by a display device of the head-mounted wearable device based on the content output state of the head-mounted wearable device. Outputting the content may include outputting the content within a display area of the head-mounted wearable device in a first content output state and delaying the output of the content in a second content output state.

[0005] In some embodiments, determining the context includes obtaining data associated with the head-mounted wearable device, the data including at least one of location data provided by the head-mounted wearable device, image data provided by an image sensor of the head-mounted wearable device, or position data and orientation data provided by a position and orientation sensor of the head-mounted wearable device. Obtaining data associated with the head-mounted wearable device may include obtaining at least one of location data, image data, position data, or orientation data from a mobile computing device paired with the head-mounted wearable device.

[0006] In some embodiments, the method executed by the computer may also include determining the applicability of the content to the context, outputting the content in response to a determination that the content is applicable to the context, and delaying the output of the content in response to a determination that the content is not applicable to the context.

[0007] In some embodiments, outputting content includes outputting the content within a specified content display area within the display area of the head-mounted wearable device. Outputting the content includes identifying a portion of the display area of the head-mounted wearable device as the specified content display area, displaying the content only within the specified content display area, designating the remaining portion of the display area of the head-mounted display device as a non-display area, and restricting the display of content in the non-display area. Designating the remaining portion of the display area of the head-mounted wearable device as a non-display area may include designating a portion of the display area of the head-mounted wearable device corresponding to one or more situational safety factors as the non-display area, and restricting the display of content in the non-display area so as to maintain the user's field of view through the non-display area.

[0008] In some embodiments, the method performed by a computer may also include detecting a change in the context in which the head-mounted wearable device is operated, and updating the content output state of the head-mounted wearable device based on the changed context. In some embodiments, the method performed by a computer may also include modifying the manner in which the content is output based on the changed context and the updated content output state. Modifying the manner in which the content is output may include at least one of simplifying the appearance of the content in response to a detected change in one or more situational safety factors, changing the display position of the content in response to a detected change in one or more situational safety factors, pausing the output of the content in response to a detected change in one or more situational safety factors, and resuming the output of the content in response to a detected change in one or more situational safety factors.

[0009] In some embodiments, the first content output state is a state in which the output of content within the specified content display area maintains the situational safety based on the context in which the head-mounted wearable device is operated, and one or more situational safety factors associated with the operation of the head-mounted wearable device. In some embodiments, the second content output state is a state in which the delay of the output of content maintains the situational safety based on the context in which the head-mounted wearable device is operated, and one or more situational safety factors associated with the operation of the head-mounted wearable device.

[0010] In another general aspect, a non-transitory computer-readable medium, when executed by at least one processor of a computing device, causes the at least one processor to determine a context associated with the operation of the head-mounted wearable device, determine the content output state of the head-mounted wearable device based on the context and one or more situational safety factors associated with the context, and store executable instructions that cause the display device of the head-mounted wearable device to output content based on the content output state of the head-mounted wearable device. Outputting the content may include outputting the content within the display area of the head-mounted wearable device in the first content output state and delaying the output of the content in the second content output state.

[0011] In some embodiments, the instructions cause at least one processor to obtain data associated with a head-mounted wearable device, the data including at least one of location data provided by at least one of the head-mounted wearable device or a mobile computing device paired with the head-mounted wearable device, image data provided by an image sensor of the head-mounted wearable device, or at least one of position data and orientation data provided by at least one of a position and orientation sensor of the head-mounted wearable device or the mobile computing device.

[0012] In some embodiments, the instructions cause at least one processor to determine the applicability of content to a context, and in response to a determination that the content is applicable to the context, cause the content to be output, and in response to a determination that the content is not applicable to the context, delay the output of the content.

[0013] In some embodiments, the instructions cause at least one processor to identify a portion of the display area of the head-mounted wearable device as a specified content display area, display content only within the specified content display area, specify the remaining portion of the display area of the head-mounted display device as a non-display area, and restrict the display of content in the non-display area.

[0014] In some embodiments, the instructions cause at least one processor to specify a portion of the display area of the head-mounted wearable device corresponding to one or more situational safety factors as a non-display area, and restrict the display of content in the non-display area so as to maintain the user's field of view through the non-display area.

[0015] In some embodiments, the instructions cause at least one processor to detect a change in the context in which the head-mounted wearable device is being operated, update the content output state of the head-mounted wearable device based on the changed context, and modify the manner in which content is output based on the changed context and the updated content output state. In some examples, the instructions cause at least one processor to modify the manner in which content is output, and modifying includes simplifying the appearance of the content in response to a detected change in one or more situational safety factors, changing the display position of the content in response to a detected change in one or more situational safety factors, pausing the output of the content in response to a detected change in one or more situational safety factors, or resuming the output of the content in response to a detected change in one or more situational safety factors.

[0016] In some embodiments, the first content output state is a state in which the output of content within the display area maintains situational safety based on the context in which the head-mounted wearable device is being operated and one or more situational safety factors associated with the operation of the head-mounted wearable device.

[0017] In some embodiments, the second content output state is a state in which the latency of the output of content maintains situational safety based on the context in which the head-mounted wearable device is being operated and one or more situational safety factors associated with the operation of the head-mounted wearable device.

[0018] Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0020] The present disclosure relates to systems and methods for improving safety in the use of wearable devices. In some examples, the systems and methods disclosed herein can improve safety in the use of wearable devices by reducing user distraction, particularly in situations where user distraction can pose a danger. In some examples, the systems and methods can selectively determine whether to output information to a user via a wearable device based on a determination of the context in which the wearable device is being operated. In some examples, the systems and methods can use the context determination to determine whether the wearable device is in a first content output state in which content can be output as soon as it becomes available, or a second content output state in which only appropriate, i.e., applicable, content is output in a specified portion of the output device and / or in a modified style and / or format, or a third content output state in which the output of the content is delayed or paused. In some examples, the systems and methods can make a determination of how and / or where to output information to the user based on a determination of the context in which the wearable device is being used. The systems and methods can adapt the output of information to the user based on the context in which the wearable device is being used and the information being output, in accordance with the embodiments described herein, to improve safety in the use of wearable devices.

[0021] Hereinafter, the systems and methods will be described with respect to safety associated with the use of a wearable device in the form of a head-mounted wearable device, such as smart glasses, having display and / or computing capabilities, for purposes of illustration and description only, in accordance with the embodiments described herein. The principles described herein can be applied to other types of wearable devices, particularly those in which selective output of information to a user can improve safety in the use of the wearable device based on the current context in which the wearable device is being used.

[0022] Figure 1 is a third - party view of a user including various mobile devices. Exemplary mobile devices include a first wearable device 100 in the form of head - mounted wearable devices or glasses such as smart glasses worn on the user's head, a second wearable device 180 in the form of earphones worn on one or both ears of the user, and a third wearable device 190 in the form of a watch such as a smartwatch worn on the user's wrist, including various exemplary wearable devices. Figure 1 includes a mobile device 200 held by the user in the form of a handheld computing device such as a smartphone. In some examples, the first wearable device 100, and / or the second wearable device 180, and / or the third wearable device 190, and / or the mobile device 200 may include output functions including a display output function and / or an audio output function. In some examples, the first wearable device 100, and / or the second wearable device 180, and / or the third wearable device 190, and / or the mobile device 200 may include computing functions and / or processing functions. In some examples, the first wearable device 100, and / or the second wearable device 180, and / or the third wearable device 190, and / or the mobile device 200 can access additional computing resources and processing resources that are available locally on a network, and / or on one of the mobile device 200, and / or the wearable devices 100, 180, 190. In some examples, the wearable devices 100, 180, 190 and the mobile device 200 can communicate with each other and / or with additional resources to perform information exchange, input and / or output transmission and reception, etc. The principles described herein may apply to other types of wearable devices and mobile devices not explicitly shown in Figure 1.

[0023] The following systems and methods are described with respect to improvements in safety associated with the operation of a first exemplary wearable device 100 in different contextual situations, for purposes of explanation and illustration, alone or in conjunction with an exemplary mobile device 200. The principles described herein may be applied, alone or in conjunction with an exemplary mobile device 200 and / or other types of mobile devices, to achieve improvements in safety associated with the operation of other types of wearable devices.

[0024] For purposes of explanation and illustration, an exemplary head-mounted wearable device 100 in the form of a pair of smart glasses is shown in FIGS. 2A and 2B. Specifically, FIG. 2A is a front view of an exemplary head-mounted wearable device 100, and FIG. 2B is a rear view of an exemplary head-mounted wearable device 100. The exemplary head-mounted wearable device 100 includes a frame 102 having a glass portion 107, i.e., an edge portion 103 surrounding the lens 107, and arm portions 105 coupled to respective edge portions 103. In some embodiments, the lens 107 may be a corrective / prescription lens. In some embodiments, the lens 107 may be a glass portion that does not necessarily incorporate corrective / prescription parameters. A bridge portion 109 may connect the edge portions 103 of the frame 102.

[0025] The exemplary wearable device 100 shown in FIGS. 2A and 2B includes a display device 104 that can output visual content, e.g., via an output coupler 144, such that the visual content can be visible to the user. In some examples, the output coupler 144 may substantially coincide with the lens(es) 107. In this form, the wearable device 100 may also include an audio output device 106 (e.g., one or more speakers, etc.), an illumination device 108, a sensing system 110, a control system 112, at least one processor 114, and an outward-facing image sensor 116, i.e., a camera 116.

[0026] In some examples, the display device 104 may include a see-through near-eye display. For example, the display device 104 may be configured to project light onto a portion of a teleprompter glass that functions as a beam splitter mounted at an angle (e.g., 30 to 45 degrees) from a display light source. The beam splitter may enable reflectance and transmittance values that allow light from the display light source to be partially reflected while the remaining light is transmitted. Such an optical design may enable a user to view both physical items in the world, e.g., through lens 107, adjacent to content (e.g., digital images, user interface elements, virtual content, etc.) generated by the display device 104. In some embodiments, waveguides, holograms, or other combiner optical components may be used to project content from the display device 104.

[0027] In some examples, the eye-tracking device 120 may detect and track the direction and movement of the eye's line of sight. Data captured by the eye-tracking device 120 may be processed to detect and track the direction and movement of the line of sight as user input. In some embodiments, the sensing system 110 may include various sensing devices, and the control system 112 may include various control system devices including, for example, one or more processors 114 operably coupled to components of the control system 112. In some embodiments, the control system 112 may include a communication module that provides for communication and exchange of information between the wearable device 100 and other external devices.

[0028] The exemplary wearable device 100 can include more or fewer features than those described above. The principles described herein are applicable to head-mounted wearable devices with or without corrective lenses.

[0029] FIG. 2C is a front view of an exemplary mobile device 200 shown in FIG. 1. The exemplary mobile device 200 may include an interface device 210. In some embodiments, the interface device 210 may function as an input device that includes, for example, a contact surface 212 capable of receiving touch input from a user. In some embodiments, the interface device 210 may function as an output device that includes, for example, a display portion 214 that enables the interface device 210 to output information to the user. In some embodiments, the interface device 210 may be able to function as both an input device and an output device. The exemplary mobile device 200 may include an audio output device 216, i.e., a speaker, that outputs an audio signal to the user.

[0030] Exemplary mobile device 200 may include a sensing system 220 that includes various sensing system devices. In some examples, the sensing system devices may include, for example, one or more image sensors, one or more position and / or orientation sensors, one or more audio sensors, one or more touch input sensors, and other such sensors. The exemplary mobile device 200 shown in FIG. 2C includes an image sensor 222 included in the front camera of the mobile device 200. The exemplary mobile device 200 may include additional image sensors, such as, for example, a world-facing camera. The exemplary mobile device 200 shown in FIG. 2C includes an inertial measurement unit (IMU) 224 that includes, for example, a position and / or orientation sensor, and / or an acceleration sensor such as an accelerometer, gyroscope, magnetometer, and other such sensors that can provide position and / or orientation data and / or acceleration data. The exemplary mobile device 200 shown in FIG. 2C includes an audio sensor 226 that can detect audio signals, for example, to process user input. The exemplary mobile device 200 shown in FIG. 2C includes a touch input sensor 228 that corresponds to the contact surface 212 of the interface device 210. The touch input sensor 228 can detect touch input signals for processing as user input. Exemplary mobile device 200 may include a control system 270 that includes various control system devices. Exemplary mobile device 200 may include a processor 290 to facilitate the operation of the mobile device 200. Exemplary mobile device 200 may include a communication module 250 that provides a connection to a network for communication via other devices external to the exemplary mobile device 200 (e.g., exemplary wearable device 100 and other such devices). In some examples, access to the network may be shared with external devices such as exemplary wearable device 100 via exemplary mobile device 100.

[0031] Systems and methods may provide for determining context information associated with the current use of a wearable device, according to embodiments described herein. In some examples, context information may be considered when determining whether to output content to a user or delay the output of content to the user. In some examples, context information may be considered when determining the manner in which content is output to a user. In an example where the wearable device is a head-mounted wearable device (such as wearable device 100 shown in FIGS. 1, 2A, and 2B), the context information may include, for example, whether the output of the content will cause the user to be distracted, whether the output of the content is appropriate, i.e., applicable, to the user at a particular time based on the context information, whether the output of the content can be adapted to minimize distraction from the user's situational awareness or avoid occlusion of an object in front of the user. Systems and methods may provide for selective output of content by wearable device 100 in a manner that helps the user maintain situational awareness and situational safety, according to embodiments described herein.

[0032] FIG. 3 is a block diagram of an exemplary system for implementing the systems and methods described herein. The system may include one or more wearable computing devices. An exemplary computing device 300 in the form of a wearable device (such as the head-mounted wearable device 100 shown in FIGS. 1, 2A, and 2B) is included in the exemplary system shown in FIG. 3. The computing device 300 can operate under the control of a control system 340. The computing device 300 can communicate with one or more external devices 370 (such as the mobile device 200 shown in FIGS. 1 and 2C, other wearable computing devices, other mobile computing devices, etc.) either directly (via wired communication and / or wireless communication) or via a network 305. In some embodiments, the computing device 300 includes a communication module 350 to facilitate external communication. In some embodiments, the computing device 300 includes a sensing system 310 that includes various sensing system components, such as, for example, one or more image sensors 312, one or more position / orientation sensors (s) 314 (including, for example, an inertial measurement unit, an accelerometer, a gyroscope, a magnetometer, etc.), one or more audio sensors 316 capable of detecting audio input, one or more touch input sensors 318 capable of detecting touch input, and other such sensors. The computing device 300 can include more, or fewer, sensing devices and / or combinations of sensing devices.

[0033] In some examples, the image sensor(s) 312 may include a camera, such as an outward-facing, i.e., world-facing camera, that can capture still images and / or video of the environment outside of the computing device 300, for example. The still images and / or video may be displayed by a display device of the output system 330 and / or transmitted externally via the communication module 350 and network 305 and / or stored in the memory 320 of the computing device 300. The computing device 300 may include one or more processors(s) 360. The processor 360 may include various modules or engines configured to perform various functions. In some embodiments, the processor(s) 360 may include an object classification module(s), a pattern classification module(s), a configuration identification module(s), and other such processors. The processor(s) 360 may be formed on a substrate configured to execute one or more machine-executable instructions, or portions of software, firmware, or combinations thereof. The processor(s) 360 may be semiconductor-based, including semiconductor material capable of executing digital logic. The memory 320 may include any type of storage device that stores information in a format readable and / or executable by the processor(s) 360. The memory 320 may store applications and modules that perform specific operations when executed by the processor(s) 360. In some examples, the applications and modules may be stored on an external storage device and loaded into the memory 320.

[0034] Computing device 300 can access additional resources 390 to facilitate determination of a number of factors to consider in selective output of content to a user via computing device 300. In some examples, additional resources 390 are locally available on computing device 300. In some examples, additional resources 390 are available on computing device 300 via network 305. In some examples, additional resources 390 are available on a computing device via external device 370. In some examples, some of additional resources 390 are locally available on computing device 300 and some of the additional resources are available via network 305 and / or via external device 370. Additional resources 390 can include, for example, a server computer system, a processor, a database, a memory storage, and the like. In some examples, additional resources 390 can include various processing engines and / or modules, models (including machine learning models), algorithms, and the like. In some examples, additional resources 390 can include an object classification engine and / or a pattern classification engine. In some examples, the object and / or pattern classification engine can detect and / or identify one or more features of the image data sent thereto for processing. Additional resources 390 can include one or more models and / or one or more algorithms. In some examples, the model(s) and / or algorithm(s) can be executed by a processor to determine context information associated with one or more uses of a computing device included in the system, application of rules for output of information, the format and / or style in which information is output, change in the format and / or style in which information is output, and other such processes.

[0035] As described above, in some examples, the external device 370 can communicate with the computing device 300. For example, the external device 370 can be paired with, i.e., operably coupled to, the computing device 300. In some examples, the external device 370 can provide processing resources and / or computing resources to the computing device 300. In some examples, the external device can provide additional data for use in context determination and selective output of content to the user of the computing device 300. As described above, in some examples, the external device 370 can be in the form of, for example, the mobile device 200 shown in FIGS. 1 and 2C. In some examples, the external device 370 can operate under the control system 380. The external device 370 can communicate directly (via wired communication and / or wireless communication) or via the network 305 with one or more other devices. The external device can include a communication module 382 to facilitate external communication. The external device 370 includes a sensing system 371 that includes various sensing system components, such as, for example, one or more image sensors 372, one or more position / orientation sensors 373 (e.g., including an inertial measurement unit, accelerometer, gyroscope, magnetometer, etc.), one or more audio sensors 374 capable of detecting audio input, one or more touch input sensors 375 capable of detecting touch input, and other such sensors.

[0036] The external device may include an output system 377 that includes one or more display devices capable of outputting image content and one or more audio output devices capable of outputting audio content. The external device 370 may include one or more processors 384. The processor(s) 384 may include various modules or engines configured to perform various functions. The processor(s) 384 may be formed on a substrate configured to execute one or more machine-executable instructions, or portions of software, firmware, or combinations thereof. The processor(s) 384 may be semiconductor-based, including semiconductor material capable of executing digital logic. The memory 376 may include any type of storage device that stores information in a format readable and / or executable by the processor(s) 384. The memory 376 may store applications and modules that, when executed by the processor(s) 384, perform certain operations. In some examples, the applications and modules may be stored on an external storage device and loaded into the memory 376.

[0037] The following, systems and methods are described with respect to a wearable device in the form of an exemplary head-mounted wearable device 100 shown in FIGS. 1, 2A, and 2B. In some examples, the systems and methods are described with respect to the operation of an exemplary head-mounted wearable device 100, alone or in conjunction with an external device such as the exemplary mobile device 200 shown in FIGS. 1 and 2C. The principles described herein may apply to other types of wearable devices and / or combinations of devices.

[0038] In some examples, a user wearing the exemplary head-mounted wearable device 100 may consume content output by the head-mounted wearable device 100. In some examples, the content is output in the form of image content output by the display device 104 and may be visible within the user's field of view. In some examples, the content may be in the form of audio content output by the audio output device 106. In some examples, the content may include a combination of image content and audio content. In some situations, the output of content by the head-mounted wearable device 100 may result in user distraction or occlusion of an important object in front of the user. In some situations, such distraction or occlusion may compromise the safety of the user and / or other persons in the vicinity of the user, or pose a danger to the user and / or persons in the vicinity of the user. These types of situations may require a delay in the output of information to the user, a modification of how the information is output to the user, etc. to maintain situational safety while the head-mounted wearable device 100 is in use.

[0039] For example, while a user is engaged in an activity that benefits from substantial care, the content and / or notifications displayed by the display device 104 may distract the user's attention. In an example where the user is walking in a somewhat congested setting (i.e., pedestrian traffic, obstacles in the user's path, automobile traffic, etc.), in particular, when the content and / or notifications are not appropriate for the current activity, i.e., not applicable, the content and / or notifications may cause distraction of the user and / or may compromise the safety of the user and / or persons in the vicinity of the user. Similarly, in an example where the user is driving, the display of the content and / or notifications may cause distraction that may compromise the safety of the user and / or persons in the vicinity of the user. Further, the display of content and / or notifications while the user is engaged in an activity such as walking or driving may, in some situations, block portions of the user's field of vision and thus pose a potential safety hazard. A number of other types of environmental conditions that benefit from reduced distraction of the user may similarly require a delay in the output of information to the user, modification of how the information is output to the user, etc. Such environmental conditions may include, for example, icy sidewalks and / or roads, precipitation that impairs visibility, and other such conditions.

[0040] According to embodiments described herein, a system and method can determine a context associated with the use of a wearable device and, based on the determined context, make a decision as to whether to output information or delay the output of information via a mobile device. When deciding to output information, the system and method can determine the manner in which the information is to be output based on the context associated with the use of the wearable device, in accordance with embodiments described herein. In the case of image-based, i.e., visual, output, the manner in which the information is output can include, for example, determining the visual region in which the information is output in order to minimize distraction and avoid occlusion of important objects within the user's field of view. In some examples, this can include modifying the content being output, such as integrating and / or simplifying the information being output, in order to reduce the size and / or level of complexity of the information being output. In some examples, this can include delaying the output of information until the context in which the wearable device is being used changes and the output of the information no longer causes distraction or occlusion that could pose a safety issue for the user.

[0041] Figure 4A is a third-party view of a user within the surrounding environment 400. In the example shown in Figure 4A, the user is wearing a head-mounted wearable device 100 (not shown in Figure 4A). A handheld mobile computing device 200 (not shown in Figure 4A) can be carried by the user. In some examples, the wearable device 100 and the mobile device 200 can be paired, i.e., operatively coupled. When in this state, the wearable device 100 and the mobile device 200 can communicate with each other, exchange data and information, share computing resources, etc. In some examples, the wearable device 100 and the mobile device 200 can operate substantially independently. Figures 4B - 4D are third-party views of the user's field of view of the surrounding environment 400 as seen through one of the lenses 107 of the exemplary wearable device 100. In particular, Figures 4B - 4D show the user's field of view 405 of the surrounding environment 400 through one of the lenses 107 of the head-mounted wearable device 100. In this example, the user's field of view can represent what is visible to the user when the user looks through the lens 107 of the head-mounted wearable device 100 at the surrounding environment 400, for example. The user's field of view changes, for example, when the user changes / shifts the head position, moves, etc., causing corresponding changes to what is visible to the user. In this example, Figures 4B - 4D show the user's field of view 405 through one of the lenses 107 of the head-mounted wearable device 100 as the user walks along the sidewalk 410 towards the crosswalk 420 and then turns to cross the crosswalk 420.

[0042] In the user's field of view 405 shown in FIG. 4B, the user is walking along the sidewalk 410 towards the crosswalk 420. In FIG. 4B, the content in the form of the virtual object 450 output by the display device 104 is displayed to the user to provide route guidance. In the example shown in FIG. 4B, the virtual object 450 includes relatively detailed route guidance such as text instructions, estimated arrival times, streets, and direction indications. In this example, the virtual object 450 including relatively detailed route guidance may be displayed in response to a determination that the user is in a situation where the output of the virtual object 450 including detailed route guidance and any distraction associated with the output of the virtual object 450 will not compromise situational safety. This determination may be based on safety factors of various different situations and / or combinations of safety factors of the situation. These factors may include, for example, detecting that there is little to no pedestrian traffic on the sidewalk 410, that there are no imminent turns in the route guidance provided by the virtual object 450, that no automobile traffic is detected on the user's imminent route, that there are no detected obstacles that could pose a danger on the relatively clear route / user's route, and other such factors. The example shown in FIG. 4B represents the content output state of the wearable device 100 where the environmental conditions in which the wearable device 100 operates enable the output of content in a relatively relaxed manner.

[0043] In the user's field of view 405 shown in FIG. 4C, the user is substantially at the same position on the sidewalk 410 as in FIG. 4B. However, in FIG. 4C, one or more obstacles, i.e., obstructions, are detected in the user's path. In the example shown in FIG. 4C, exemplary obstacles include pedestrian traffic on the sidewalk within the user's path and a trash container on the sidewalk within the user's path. In this situation, the virtual object 451 is output by the display device 104. The virtual object 451 shown in FIG. 4C includes a modified, i.e., simplified, route guidance. Since the virtual object 451 shown in FIG. 4C is somewhat transparent, the user's field of view 405 of the surrounding environment 400 remains somewhat visible to the user through the virtual object 451. The modified, i.e., simplified, route guidance provided by the virtual object 451 can reduce distraction caused by the display of the virtual object 451 and allows the user to maintain situation awareness and avoid obstacles along the route while maintaining recognition of the route to follow according to the route guidance. Thereby, the situational safety during the use of the wearable device 100 can be improved. The example shown in FIG. 4C represents the content output state of the wearable device 100 where the environmental conditions under which the wearable device 100 operates require changes and / or additional controls in the manner in which the content is output.

[0044] In some examples, the system may identify one or more areas or zones within the user's field of view 405 of the surrounding environment 400. Based on the context associated with the use of the wearable device 100, obstacles, hazards, and other items to consider, the system may identify, for example, one or more of the areas or zones within the user's field of view 405 where the output of information in the form of a virtual object 451 may be displayed. In FIG. 4C, three exemplary content display areas 480 are identified. The exemplary content display areas 480 may correspond to areas, i.e., portions, of the user's field of view 405 where the display of the virtual object 451 is less likely to cause distraction to the user of the wearable device 100 and / or is less likely to block or interfere with the user's view of areas where safety is important, thus enabling the user to maintain situation awareness / situation safety while using the wearable device 100.

[0045] In the user's field of view 405 shown in FIG. 4C, the user is substantially at the same position on the sidewalk 410 as in FIG. 4B. However, in FIG. 4C, one or more obstacles, i.e., obstructions, are detected in the user's path. In the example shown in FIG. 4C, the exemplary obstacles include pedestrian traffic on the sidewalk within the user's path and a trash container on the sidewalk within the user's path. In this situation, the virtual object 451 is output by the display device 104. The virtual object 451 shown in FIG. 4C includes a modified, i.e., simplified, route guidance. Since the virtual object 451 shown in FIG. 4C is somewhat transparent, the user's field of view 405 of the surrounding environment 400 remains somewhat visible to the user through the virtual object 451. The modified, i.e., simplified, route guidance provided by the virtual object 451 may reduce distraction caused by the display of the virtual object 451 and enable the user to maintain situation awareness and avoid obstacles along the route while maintaining recognition of the route to follow according to the route guidance. Thereby, the situation safety during the use of the wearable device 100 may be improved.

[0046] As described above, in some examples, the determination of whether to output information, and / or in what manner to output information, and / or at what position within the user's field of view to output information can be determined based on a situational safety factor associated with the use of the wearable device 100. In some examples, these types of factors can be detected in the image data captured by the image sensor 116 of the head-mounted wearable device 100, for example. The image data can include portions of the surrounding environment 400 within the user's path, areas surrounding the user, etc., which can provide an indication of the situational safety factor to be considered. The image data can be processed by, for example, a classification engine (one or more classification engines (s) included in the additional resource 390 described above with respect to FIG. 3, etc.). The objects and / or patterns detected in the image data can be processed by, for example, one or more models (s), algorithms (s), etc. to determine the context in which the wearable device 100 is currently operating. In some examples, the determination of context can include, for example, the determination of location. This can include the processing of location data provided by the wearable device 100 and / or the mobile device 200 paired with the wearable device 100. In some examples, the determination of context can include the processing of location and / or orientation data provided by the corresponding sensors of the wearable device 100 and / or the mobile device 200. In some examples, the location and / or orientation data can represent the user's movement corresponding to, for example, checking traffic conditions, changing direction, starting / stopping movement, using a rearview or side mirror, etc.

[0047] In some examples, the processing of the detected object and / or pattern and / or location may provide an indication of the environment in which the wearable device 100 is currently operating, the activity the user is engaged in, etc., to determine the context. In some examples, the processing of the detected object and / or pattern and / or location may identify potential hazards in the vicinity of the user. In some examples, the processing of the detected object and / or pattern and / or location and the corresponding context may be used to determine the placement of the output of information within the user's field of view 405. In some examples, the processing of the detected object and / or pattern and / or location and the corresponding context may be used to determine the area within the user's field of view 405 in which information (e.g., in the form of virtual objects) can be presented to the user without causing a hazard. In some examples, the processing of the detected object and / or pattern and / or location and the corresponding context may be used to determine the area within the user's field of view 405 in which the presentation of information (e.g., in the form of virtual objects) may pose a hazard to the user and / or to people in the vicinity of the user.

[0048] In the user's field of view 405 shown in FIG. 4D, the user reached the corner of the sidewalk 410 and turned to face the crosswalk 420 in preparation for crossing the roadway. In FIG. 4D, the system detected that the user had arrived at the crosswalk 420. The system can detect the user's position on the crosswalk 420 prepared for crossing the road and can pause the display of information (e.g., in the form of virtual objects 450, 451). The position of the user on the crosswalk 420 can be detected based on the processing and analysis of, for example, image data acquired by the image sensor 116 of the wearable device 100, location data provided by the wearable device 100 and / or the mobile device 200, etc. For example, a pattern associated with the crosswalk 420 can be detected within the image data. One or more automobiles can be detected within the image data. Location data, including, for example, global positioning system data, traffic data, position and / or orientation data provided by corresponding sensors of the wearable device 100 and / or the mobile device 200, etc., can provide information related to the level of risk associated with walking and / or driving in a particular area. In the example shown in FIG. 4D, the system paused the display of information at this point to eliminate distraction in a situation where the user was determined to benefit from exercising substantial care. That is, the system paused the display of information to enable the user to focus on environmental / situation safety issues such as approaching vehicle traffic, pedestrian traffic, signals, etc. while crossing. In some examples, the system can resume the output / display of information of virtual objects (such as route guidance provided by exemplary virtual objects 450 and 451) when it is determined that situation safety considerations require the resumption of the output of information.The example shown in FIG. 4D represents a content output state of the wearable device 100 where the environmental conditions under which the wearable device 100 is operating require additional changes and / or additional control of the manner in which content is output (as compared to the content output state shown in FIG. 4C, i.e., a content output state where content is not being output by the wearable device 100 or where the output of content is temporarily paused or delayed).

[0049] FIG. 4C shows that the system can adapt or modify the display of information based on the context and surroundings in which the wearable device 100 is used. FIG. 4D shows that the system can delay or temporarily stop the output of information based on the context and ambient environment in which the wearable device 100 is used. In some examples, the system can enable the output of some types of information and delay the output of other types of information. The decision of whether to enable the output of information or delay the output of information may be associated with the context and surroundings in which the wearable device 100 is used, as well as the appropriateness, i.e., applicability, of the information being output to the context and surroundings. In the exemplary situations shown in FIGS. 4A - 4D, the system can determine that information in the form of route guidance is appropriate, i.e., applicable, to the current use of the wearable device 100 and can output that information when it is determined that the output of the information (e.g., in the form of virtual objects 450, 451) does not cause excessive distraction and does not pose a safety hazard. In the exemplary situations shown in FIGS. 4A - 4D, the system can determine that other types of information, such as notifications of incoming messages, may not be appropriate, i.e., applicable, to the current use of the mobile device 100 and / or may cause unnecessary distraction when received. In this situation, the system can delay the output of the information (i.e., the display of the notification of the incoming message) until it is determined that the output of the information does not cause excessive distraction and / or does not compromise the safety of the situation.

[0050] Figs. 5A-5E are third-party views of the field of view of a user of the exemplary wearable device 100. In particular, Figs. 5A-5E are third-party views of the user's field of view 505 of the surrounding environment 500 through one of the lenses 107 of the head-mounted wearable device 100 worn by the user. In this example, the user's field of view can represent what is visible to the user when the user looks through, for example, the lens 107 of the head-mounted wearable device 100 at the surrounding environment 500. The field of view changes, for example, when the user changes / shifts, moves, etc. the head position, causing a corresponding change to what is visible to the user. In this example, Figs. 5A-5E show the user's field of view 505 through one of the lenses 107 of the wearable device 100 while the user is seated in the driver's seat of a vehicle. In this example, the hand-held mobile computing device 200 (not shown in Figs. 5A-5E) may be with the user, for example, in a storage location inside the vehicle. In some examples, the wearable device 100 and the mobile device 200 can be paired, i.e., operably coupled. When in this state, the wearable device 100 and the mobile device 200 can communicate with each other, exchange data and information, share computing resources, etc. In some examples, the wearable device 100 and the mobile device 200 can operate substantially independently.

[0051] In some examples, the context associated with the use of the wearable device 100 may be determined based on, for example, an image captured by the image sensor 116 of the wearable device 100. In some examples, the context associated with the use of the wearable device 100 may be determined based on location data provided by the wearable device and / or the mobile device 200. In some examples, the context associated with the use of the wearable device 100 may be determined based on position and / or orientation data provided by sensors of the wearable device 100 and / or the mobile device 200. In some examples, the determination of whether to output information, and / or in what manner to output the information, and / or at which position within the user's field of view to output the information may be determined based on a situation safety factor associated with the use of the wearable device 100. In some examples, these types of factors may be detected in the image data captured by the image sensor 116 of the head-mounted wearable device 100, which may provide an indication of the situation safety factor to be considered. The image data may be processed, for example, by a recognition engine (such as one or more recognition engines (s) included in the additional resource 390 described above with respect to FIG. 3). The objects and / or patterns detected in the image data may be processed, for example, by one or more models (s), algorithms (s), etc. to determine the context in which the wearable device 100 is currently operating. In some examples, the determination of the context may include processing of location data provided by the wearable device 100 and / or the mobile device 200. In some examples, the determination of the context may include processing of position and / or orientation data provided by corresponding sensors of the wearable device 100 and / or the mobile device 200, which may represent, for example, the user's movement in response to checking traffic conditions, changing direction, starting / stopping movement, etc.

[0052] For example, the processing of the image data acquired by the image sensor 116 of the wearable device 100 can detect objects such as the steering wheel 510, the rearview mirror 520, the dashboard 530, the instrument cluster 540, the boundary 550 of the windshield 555, and other such objects. The detection position and / or relative position of these objects in the image data can provide an indication that the user of the wearable device is in the driver's seat of the vehicle. Similarly, for example, objects such as the boundary 560 of the road 565, the boundary line 562, etc. can be detected. The processing of the location information provided by the wearable device 100 and / or the mobile device 200 may indicate that the vehicle is moving and can provide information related to the road and / or traffic conditions, approaching hazards, etc. The processing of the position and / or orientation sensors of the wearable device 100 and / or the mobile device 200 can provide an indication of the position and / or orientation and / or change in the line of sight of the user within the vehicle. This data can be used to enhance the understanding of the context of the environment in which the wearable device 100 is operated. Therefore, the objects and / or patterns detected in the image data, and / or location data, and / or position / orientation data provided by the sensors of the wearable device 100 and / or the mobile device 200 can provide an indication of the environment in which the wearable device 100 is currently operating, the activity the user is engaged in, etc. for determining the context. For example, in the arrangements shown in FIGS. 5A - 5E, based on the analysis of the data, it can be determined that the user of the wearable device 100 is in the driver's seat of the vehicle and is the operator of the vehicle.

[0053] In some examples, image data and / or location data and / or position / orientation data may be collected and processed substantially continuously to identify potential hazards in the vicinity of the user, particularly as the vehicle moves. In some examples, the processing of detected objects and / or patterns and / or locations and / or positions / orientations, and corresponding contexts, may be used to determine the placement of information output within the user's field of view 505. In some examples, the processing of detected objects and / or patterns and / or locations and / or positions / orientations and corresponding contexts may be used to determine the area within the user's field of view 505 in which information (e.g., in the form of virtual objects) can be presented to the user without causing an excessive level of distraction and without posing a danger. In some examples, the processing of detected objects and / or patterns and / or locations and / or position / orientation data and corresponding contexts may be used to determine the area within the user's field of view 505 in which the presentation of information (e.g., in the form of virtual objects) may pose a danger to the user and / or to people in the vicinity of the user. For example, the processing of detected objects and / or patterns and / or locations and / or position / orientation data and corresponding contexts may be used to determine the area within the user's field of view in which the presentation of information does not block portions of the user's field of view 505 and enables continued safe use of the wearable device 100. In the example of driving shown in FIGS. 5A - 5E, this type of data and analysis may provide for the determination of the area within the user's field of view 505 in which information can be presented while allowing the driver / user of the wearable device 100 to maintain situation awareness and situational safety.

[0054] FIG. 5B shows an exemplary content display area 580. The exemplary content display area 580 may define an area, or zone, or portion of the user's field of view 505 where the display of information is less likely to cause distraction to the user of the wearable device 100. The exemplary content display area 580 may correspond to an area, or zone, or portion of the user's field of view 505 that enables the user to maintain situational awareness and situational safety while using the wearable device 100 because the output of information (e.g., in the form of the display of virtual objects) does not block or obstruct the user's field of view in areas where safety is critical. One exemplary content display area 580 is shown in FIG. 5B for purposes of illustration and description. FIG. 5C shows the output of information in the form of virtual object 570 within the designated content display area 580.

[0055] In the exemplary arrangements shown in FIGS. 5B and 5C, the content display area 580 corresponds to a portion of the dashboard 530. For example, the output of information in the form of the display of virtual object 570 within the designated content display area 580 enables the user's field of view through the front windshield 555, including the peripheral portion of the front windshield 555, to remain clear and not be blocked by the output of information. Similarly, by displaying the virtual object 570 within the designated content display area 580, the user's field of view of the rearview mirror 520 and the user's field of view of the instrument cluster 540 can remain clear and not be blocked by the output of the virtual object 570. In some examples, one or more non-content display areas may be designated within the user's field of view 505. The non-content display area(s) may correspond to portions of the user's field of view 505 that remain clear and unobstructed. Thus, in the example shown in FIG. 5B, the portion of the user's field of view 505 corresponding to the front windshield 555 may be designated as a first non-content display area 559. The portion of the user's field of view 505 corresponding to the rearview mirror 520 may be designated as a second non-content display area 529. The portion of the user's field of view 505 corresponding to the instrument cluster 540 may be designated as a third non-content display area 549.

[0056] Output of information, for example, in the form of the virtual object 570 within the specified content display area 580 can reduce the distraction of the user caused by the output of the information. By outputting information to the specified content display area 580, it can be possible to keep the remaining parts of the user's visual field 505 (i.e., the non-content display areas 529, 549, 559) clear and visible to the user of the wearable device 100. Thereby, the user may be able to maintain situation awareness and situation safety while using the wearable device 100 in this situation.

[0057] In the example shown in FIG. 5C, the content in the form of the virtual object 570 output by the display device 104 is displayed to the user within the content display area 580 of the user's visual field 505. In this example, the content in the form of the virtual object 570 provides route guidance. In the example shown in FIG. 5C, the virtual object 570 includes relatively detailed route guidance such as text instructions, exit numbers, estimated arrival times, and direction indications. In this example, the virtual object 570 including relatively detailed route guidance may be displayed in response to the user's determination that there is no situation where the output of the virtual object 570 including detailed route guidance and any distraction associated with the output of the virtual object 570 can compromise situational safety. This determination may be based on safety factors of various different situations and / or combinations of safety factors of the situation. In this particular example, these factors may include, for example, detecting that there is little to no other vehicle traffic in the vicinity of the user, a relatively straight driving path / route ahead, no imminent turns / exits in the route guidance provided by the virtual object 570, no approaching stops, obstacles, dangers, and / or other such factors. In some examples, these factors may be determined based on, for example, location data provided by the wearable device 100 and / or a mobile device 200 disposed at the same location as the wearable device 100 and communicating with the wearable device 100, image data acquired by the image sensor 116 of the wearable device 100, and other such data. In the examples shown in FIGS. 5A-5C, the content output state of the wearable device 100 instructs a controlled output only of content that is appropriate, i.e., applicable, to the current context (i.e., driving) based on the context and environmental conditions in which the wearable device 100 is operating, and only within the specified content display area 580.

[0058] In the user's field of view 505 shown in FIG. 5D, the user is at a similar position on the road as in FIG. 5C. However, in FIG. 5D, one or more obstacles, i.e., obstructions, are detected in the user's path. That is, in the example shown in FIG. 5D, the traffic volume on the road is higher, and the user may need to be more concentrated and attentive. In this situation, the virtual object 571 is output by the display device 104 and is displayed within the content display area 580. The virtual object 571 shown in FIG. 5D includes a modified, i.e., simplified, route guidance. Since the virtual object 571 shown in FIG. 5D is somewhat transparent, the user's field of view 505 of the surrounding environment 500 remains somewhat visible to the user through the virtual object 571. The modified, i.e., simplified, route guidance provided by the virtual object 571 can reduce distraction caused by the display of the virtual object 571 and enables the user to maintain situational awareness when passing through the intersection while maintaining the recognition of the route to follow according to the route guidance. Thereby, the situational safety during the use of the wearable device 100 in the situation of this example can be improved. In the example shown in FIG. 5D, the content output state of the wearable device 100 may be based on a change in context and a change in the environmental conditions in which the wearable device 100 is operating, and thus (compared to the content output state shown in FIG. 5C) requires additional changes and / or additional control in the manner in which the content is output.

[0059] As described above, in some examples, the determination of whether to output information, and / or in what manner to output information, and / or at which position within the user's field of view to output information, may be determined based on the determination of the context of use of the wearable device 100 and the situational safety factor associated with the use of the wearable device 100 in the determined context. In the examples described with respect to FIGS. 5A-5E, location data (e.g., GPS data, associated traffic data, route settings, etc.) accessible via the wearable device 100 and / or by a mobile device 200 paired with the wearable device 100 may provide at least a portion of the data used by the model(s) and / or algorithm(s) when making decisions regarding the output of information. In some examples, the context and the determination of whether to output information may include the processing of location and / or orientation data provided by corresponding sensors of the wearable device 100 and / or the mobile device 200. In some examples, the location and / or orientation data may represent, for example, the user's movement corresponding to checking traffic conditions, changing direction, starting / stopping movement, etc. In some examples, these types of factors may be detected, for example, when image data captured by the image sensor 116 of the head-mounted wearable device 100 captures the area surrounding the user. The location data and / or location / orientation data and / or image data may be processed, for example, by the model(s) and / or algorithm(s) (e.g., available via the additional resource 390 described above with respect to FIG. 3).

[0060] In the user's field of view 505 shown in FIG. 5E, the user turns their head (temporarily) sideways from the windshield 555 towards the vehicle's side window and side mirror 525. This movement may indicate, for example, that the user is performing a visual check of a blind spot, a visual check before changing lanes, etc. In FIG. 5E, the system detects this movement and determines that the user's attention may be (temporarily) directed towards a specific task, and may pause the display of information (e.g., in the form of virtual objects 570, 571). That is, to provide the user with concentration on the current task (i.e., in this example, checking the blind spot, changing lanes), the system (temporarily) pauses the output of information (i.e., the display of virtual objects 570, 571). This can reduce distraction when the user is performing the current task and ensure that the user's field of view is not blocked or obstructed for the output of information during the task. In the example shown in FIG. 5E, the content output state of the wearable device 100 may be based on changes in context and environmental conditions in which the wearable device 100 is operating, and thus requires additional changes and / or additional control of the manner in which content is output (compared to the content output state shown in FIG. 5D).

[0061] In some examples, the determination of this type of shift in the user's focus may be determined based on data provided by the position and / or orientation sensor(s) of the wearable device 100, and / or the position and / or orientation sensor(s) of the mobile device 200, and / or image data provided by the image sensor 116 of the wearable device 100. In some examples, the determination of the shift in focus (based on a change in head orientation) may be made based on data provided by the position and / or orientation sensor(s) of the wearable device 100 as compared to position / orientation data associated with the mobile device 200. In this example, the position / orientation data associated with the mobile device 200 may correspond to the position / orientation of the vehicle and thus may define a baseline position / orientation. The difference in the position / orientation of the data from the wearable device 100 from that baseline may confirm (in this example) that the user has turned their head as shown in FIG. 5E. By comparing with the movement of the baseline provided by the mobile device 200, the system may be able to distinguish between an actual intentional rotation of the user's head and a change in the position and / or orientation of the user's head (where the change in the position / orientation of the wearable device 100 would correspond to the change in the position / orientation of the mobile device 200) due to, for example, a rotation of the vehicle. In some examples, the determination of the shift in focus (based on a change in head orientation) may be made based on data provided by the position and / or orientation sensor(s) of the wearable device 100 together with image data acquired by the image sensor 116 of the wearable device 100. That is, an object, pattern, etc. detected in the image data (e.g., the detection of the side mirror 525 in the image data in this example) may be used to verify that the user has (temporarily) rotated their head to perform a blind spot check, lane change, etc.

[0062] In some examples, the determination that the user returned their head again to refocus through the windshield 555 (i.e., shifted back from the position shown in FIG. 5E to the position shown in FIG. 5C or FIG. 5D) can be made based on the data provided by the position and / or orientation sensor(s) of the wearable device 100 and / or the position and / or orientation sensor(s) of the mobile device 200, and / or the image data provided by the image sensor 116 of the wearable device 100, as described above. Upon detecting this refocus to the front of the vehicle through the windshield 555, the system can output the information again, as shown in FIGS. 5C and / or 5D.

[0063] FIGS. 5D and 5E show that the system can adapt or modify the display of information based on the context and surroundings in which the wearable device 100 is used. FIGS. 5D and 5E show that the system can delay or temporarily stop the output of information based on the context and surrounding environment in which the wearable device 100 is used. In some examples, the system can enable the output of some types of information and delay the output of other types of information. The determination of whether to enable the output of information or delay the output of information may be associated with the context and surroundings in which the wearable device 100 is used, as well as the appropriateness, i.e., applicability, of the information being output to the context and surroundings. In the exemplary situations shown in FIGS. 5A - 5E, the system can determine that information in the form of route guidance is appropriate, i.e., applicable, to the current use of the wearable device 100, and can output that information when it is determined that the output of the information (e.g., in the form of virtual objects 570, 571) does not cause excessive distraction and / or pose a safety hazard.

[0064] In the exemplary situations shown in FIGS. 5A-5E, the system may determine that other types of information, such as notifications of incoming messages, may not be appropriate, i.e., applicable, for the current use of the mobile device 100 and / or may cause unnecessary distraction upon receipt. In this situation, the system may delay the output of the information (i.e., the display of the notification of the incoming message) until it is determined that the output of the information does not cause excessive distraction and / or does not compromise situational safety. For example, the system may determine that the vehicle has stopped based on, for example, location data provided by the wearable device 100 and / or the mobile device 200, and / or position and / or orientation data provided by the wearable device and / or the mobile device 200, and may output other types of information (notifications, updates, information requested by the user, etc.) until it is determined that the vehicle is moving again. The system may determine that the user of the wearable device 100 is in the passenger seat (rather than the driver's seat) based on, for example, image data provided by the wearable device 100, and may output information to the user according to different parameters than when the user is in the driver's seat.

[0065] FIGS. 4A-4D illustrate the application of the principles described herein to the use of an exemplary head-mounted wearable device 100 during walking. FIGS. 5A-5E illustrate the application of the principles described herein to the use of an exemplary head-mounted display device 100 during driving. The principles described herein may be applied to the use of an exemplary head-mounted wearable device 100 in other environments where the user experience can be improved and situational safety can be enhanced by reducing distraction. Similarly, the principles described herein may be applied to the use of other types of wearable devices in the exemplary environments described and in other environments where the user experience can be improved and situational safety can be enhanced by reducing the user's distraction.

[0066] Figures 4A - 4D and 5A - 5E illustrate that the output of virtual content is modified and / or (temporarily) suspended based on the analysis of the detected safety factors and considerations in the specific context in which the wearable device 100 is used. In the examples shown in Figures 4A - 5E, the output of the content is modified, for example, simplified or temporarily suspended, in order to reduce distraction in situations where situational safety considerations require such modification / suspension of the output of visual content. In some examples, visual content can be replaced by audible content in order to reduce distraction that may be caused by the output of visual content in a particular situation. For example, in situations where the safety of the user's field of vision may be obstructed by important features even in a modified / simplified mode of output of visual content, audible content can be output instead of visual content.

[0067] Figure 6 is a flowchart of an exemplary method 600 for managing the output of content within a wearable device, according to an embodiment described herein. As described above, content can be output to a user by a wearable device, such as the head-mounted wearable device 100 described above, based on the context associated with the use of the wearable device in order to reduce distraction and to maintain situational awareness and situational safety during use of the wearable device. The exemplary method 600 shows operations that are performed in order for purposes of illustration and description. The operations can be performed in an order different from that shown, or in parallel or overlapping fashion.

[0068] Data associated with the wearable device can be collected when the wearable device is operated (block 610). The data can include, for example, location data associated with the wearable device, position and / or orientation data associated with the wearable device, image data obtained by an image sensor of the wearable device, a capture of an image of the environment in which the wearable device is operated, and other such data. If a mobile device such as the handheld mobile computing device 200 described above is determined to be paired with, i.e., operably coupled to, the wearable device (block 615), data associated with the mobile device can also be collected (block 620). The data can similarly include, for example, location data associated with the mobile device, position and / or orientation data associated with the mobile device, image data obtained by an image sensor of the mobile device, and other such data.

[0069] The context associated with the use of a wearable device can be determined (block 625) based on data associated with the wearable device, alone or in combination with data associated with a mobile device. The context can provide, for example, a characterization (e.g., identification and / or classification) of the environment in which the wearable device is operating, and / or the current mode of operation of the wearable device. A safety factor to be considered in the operation of the wearable device in the current environment is associated with the context. In response to a determination that content is available for output by the wearable device (block 630) and that the content is appropriate, i.e., applicable, to the determined context (block 635), the content can be output by the wearable device (block 645). This corresponds, for example, to a first content output state. The content can be output within an acceptable, i.e., specified, content display area of the wearable device, thereby enabling output of the content with minimal distraction to the user and minimal occlusion of the user's field of view of the surrounding environment.

[0070] In response to a determination that the content is not appropriate for the determined context, the output of the content can be delayed (block 640) to minimize unnecessary distraction. This corresponds, for example, to a second content output state. The output of the content can be delayed for a predetermined period, for a period determined by the wearable device and / or mobile device, and / or until it changes to a first content output state. In response to a detected change in the context (block 650), the form and / or manner in which the content is output can be modified (block 655). The detected change in the content may indicate a change in the user's focus, facilitating the need for a change in how the content is output. Modifications to the form and / or manner in which the content is output can include visual changes such as how the content is output, the level of detail included in the output of the content, a change in the output position of the content, a pause in the output of the content, etc. The process can be repeated until the usage session of the wearable device ends (block 660).

[0071] FIG. 7 shows an example of a computer device 700 and a mobile computer device 750 that can be used with the technology described herein. The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 connected to the memory 704 and a high-speed expansion port 710, and a low-speed interface 712 connected to a low-speed bus 714 and the storage device 706. Each component 702, 704, 706, 708, 710, and 712 is interconnected using various buses and may be attached to a common motherboard or attached in other manners as needed. The processor 702 processes instructions for execution within the computing device 700, including instructions stored in the memory 704 or the storage device 706, and can display graphical information of a GUI on an external input / output device such as a display 716 connected to the high-speed interface 708. In other embodiments, multiple memories and types of memories may be used, along with multiple processors and / or multiple buses as needed. Also, multiple computing devices 700 may be connected such that each device provides a portion of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0072] The memory 704 stores information within the computing device 700. In one embodiment, the memory 704 is a volatile memory unit(s). In another embodiment, the memory 704 is a non-volatile memory unit(s). The memory 704 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0073] The storage device 706 can provide large-capacity storage to the computing device 700. In one embodiment, the storage device 706 is a computer-readable medium such as, but not limited to, a floppy (registered trademark) disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configured device, or can include such a medium. The computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more of the methods described above. The information carrier is a computer-readable medium or a machine-readable medium, examples of which include the memory 704, the storage device 706, or the memory on the processor 702.

[0074] The high-speed controller 708 manages the bandwidth-intensive operations of the computing device 700, and the low-speed controller 712 manages the low-bandwidth-intensive operations. Such an assignment of functions is merely an example. In one embodiment, the high-speed controller 708 is coupled to a high-speed expansion port 710 that can receive the memory 704, the display 716 (e.g., via a graphics processor or accelerator), and various expansion cards (not shown). In an embodiment, the low-speed controller 712 is coupled to the storage device 706 and a low-speed expansion port 714. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth (registered trademark), Ethernet (registered trademark), wireless Ethernet). The low-speed expansion port may be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, etc., or a network device such as a switch or a router, for example, via a network adapter.

[0075] Computing device 700 may be implemented in a plurality of different forms as shown in the figures. For example, it may be implemented as a standard server 720, or may be run multiple times in a group of such servers. Additionally, it may be run as part of a rack server system 724. Further, it may be run on a personal computer such as a laptop computer 722. Alternatively, the components of computing device 700 may be combined with other components within a mobile device (not shown) such as device 750. Each of such devices may include one or more of computing devices 700, 750, and the entire system may be composed of a plurality of computing devices 700, 750 that communicate with each other.

[0076] Computing device 750 includes, among other components, a processor 752, a memory 764, an input / output device such as a display 754, a communication interface 766, and a transceiver 768. Additionally, a storage device such as a microdrive or other device may be provided in device 750 to provide additional storage. Each of components 750, 752, 764, 754, 766, and 768 are interconnected using various buses, and some of the components may be attached to a common motherboard or attached in other manners as required.

[0077] Processor 752 can execute instructions within computing device 750, including instructions stored in memory 764. The processor may be implemented as a chipset of chips including a plurality of separate analog and digital processors. The processor may, for example, provide coordination of other components of device 750, examples of which include control of the user interface, applications executed by device 750, and wireless communication by device 750.

[0078] The processor 752 may communicate with the user via a control interface 758 and a display interface 756 coupled to a display 754. The display 754 may be, for example, a thin film transistor liquid crystal display (TFT LCD), a light emitting diode (LED) or an organic light emitting diode (OLED) display, or other suitable display technology. The display interface 756 may include appropriate circuitry for driving the display 754 to present graphic and other information to the user. The control interface 758 may receive commands from the user and convert them for submission to the processor 752. Additionally, an external interface 762 may be provided for communicating with the processor 752 to enable short-range communication between the device 750 and other devices. The external interface 762 may provide, for example, wired communication in some embodiments, or wireless communication in other embodiments, and may use multiple interfaces.

[0079] Memory 764 stores information within computing device 750. Memory 764 can be implemented as one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. Additionally, extended memory 774 may be provided and connected to device 750 via expansion interface 772. Expansion interface 772 may include, for example, a single in-line memory module (SIMM) card interface. Such extended memory 774 may provide temporary storage space for device 750, or may store applications or other information for device 750. Specifically, extended memory 774 may include instructions for performing or supplementing the aforementioned processes, and may also include secure information. Thus, for example, extended memory 774 may be provided as a security module for device 750 and programmed with instructions that enable secure use of device 750. Additionally, secure applications may be provided via the SIMM card, along with additional information such as specific information arranged in a non-hackable manner on the SIMM card.

[0080] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more of the methods as described above. The information carrier is a computer-readable medium or a machine-readable medium, examples of which include memory 764, extended memory 774, or memory on processor 752. This information carrier may be received, for example, via transceiver 768 or external interface 762.

[0081] Device 750 may perform wireless communication via communication interface 766, which may include a digital signal processing circuit if necessary. Communication interface 766 may provide communication in various modes or protocols, examples of which include, among others, GSM (registered trademark) voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA (registered trademark), CDMA2000 or GPRS. Such communication may be performed, for example, via radio frequency transceiver 768. Additionally, short-range communication may be performed, for example, by using Bluetooth, WiFi, or other such transceivers (not shown). Additionally, a Global Positioning System (GPS) receiver module 770 may provide additional wireless data related to navigation and location to device 750, which may be used as needed by applications running on device 750.

[0082] Device 750 may also perform voice communication using audio codec 760. This audio codec 760 may receive voice information from a user and convert it into usable digital information. Similarly, audio codec 760 may generate sounds audible to the user, for example, through a speaker (such as within the handset of device 750). Such sounds may include sounds from a voice telephone call, may include recorded sounds (such as voice messages, music files, etc.), and may also include sounds generated by applications operating on device 750.

[0083] As shown in the figure, computing device 750 may be implemented in a plurality of various forms. For example, computing device 750 may be implemented as a mobile phone 780. Also, computing device 750 may be implemented as part of a smartphone 782, a personal digital assistant, or other similar mobile devices.

[0084] The various embodiments of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include execution in one or more computer programs executable and / or interpretable in a programmable system including at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from, and to transmit data and instructions to, a storage system.

[0085] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be executed in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0086] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (LED (light-emitting diode), OLED (organic LED), or LCD (liquid crystal display) monitor / screen) for displaying information to the user, as well as a keyboard and a pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and the input from the user can be received in any form including acoustic, speech language, or tactile input.

[0087] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or includes middleware components (e.g., an application server), or includes front-end components (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or in a computing device that includes any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), and the Internet.

[0088] A computing system can include a client and a server. The client and the server are generally far apart from each other and usually interact through a communication network. The relationship between the client and the server is created by computer programs that are executed on respective computers and have the relationship between the client and the server with each other.

[0089] In some embodiments, the computing device shown in the figure can include sensors that interface with an AR headset / HMD device 790 to generate an extended environment for viewing content inserted into the physical space. For example, one or more sensors included in the computing device 750 or other computing devices shown in the figure can provide input to the AR headset 790 or, generally, provide input to the AR space. The sensors can include, but are not limited to, a touch screen, an accelerometer, a gyroscope, a pressure sensor, a biometric sensor, a temperature sensor, a humidity sensor, and an ambient light sensor. The computing device 750 can use the sensors to determine the absolute position and / or the detected rotation of the computing device within the AR space, which can then be used as input to the AR space. For example, the computing device 750 can be incorporated into the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, etc. When incorporated into the AR space, by positioning the computing device / virtual object, the user can be enabled to position the computing device in a specific manner within the AR space to view the virtual object. For example, if the virtual object represents a laser printer, the user can operate the computing device as if it were an actual laser printer. The user can move the computing device left and right, up and down, draw a circle, etc., and use the device in a manner similar to using a laser printer. In some embodiments, the user can use a virtual laser pointer to aim at a target position.

[0090] In some embodiments, one or more input devices included on or connected to computing device 750 can be used as input to the AR space. The input devices can include, but are not limited to, a touch screen, keyboard, one or more buttons, trackpad, touchpad, pointing device, mouse, trackball, joystick, camera, microphone, earphone or earphone with input functionality, game controller, or other connectable input devices. A user interacting with the input devices included on computing device 750 when the computing device is incorporated in the AR space can be made to perform a particular action in the AR space.

[0091] In some embodiments, the touch screen of computing device 750 can be rendered as a touchpad within the AR space. The user can interact with the touch screen of computing device 750. The interaction is rendered within the AR headset 790, for example, as movement on a touchpad rendered within the AR space. The rendered movement can be used to control virtual objects within the AR space.

[0092] In some embodiments, one or more output devices included on computing device 750 can provide output and / or feedback to the user of AR headset 790 within the AR space. The output and feedback may be visual, tactile, or auditory. The output and / or feedback can include, but is not limited to, vibrating, turning one or more lights or strobes on and off or flashing and / or popping, sounding an alarm, chiming, playing music, and playing an audio file. The output devices can include, but are not limited to, a vibration motor, a vibration coil, a piezoelectric device, an electrostatic device, a light-emitting diode (LED), a strobe, and a speaker.

[0093] In some embodiments, computing device 750 can be displayed as another object within a computer-generated 3D environment. Interaction of the user with computing device 750 (e.g., rotating, shaking, touching, swiping a finger across the touch screen) can be interpreted as interaction with an object within the AR space. In an example of a laser pointer in the AR space, computing device 750 is displayed as a virtual laser pointer within a computer-generated 3D environment. As the user operates computing device 750, the user within the AR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 750 within the AR environment on computing device 750 or on AR headset 790. The user's interaction with the computing device can be translated into interaction with a user interface generated within the AR environment for a controllable device.

[0094] In some embodiments, computing device 750 may include a touch screen. For example, a user can interact with the touch screen to interact with a user interface for a controllable device. For example, the touch screen may include user interface elements such as sliders that can control the characteristics of a controllable device.

[0095] Computing device 700 is intended to represent various forms of digital computers and devices including, but not limited to, laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 750 is intended to represent various forms of mobile devices such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended only as examples and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0096] Numerous embodiments have been described. Nevertheless, it is understood that various modifications can be made without departing from the spirit and scope of this specification.

[0097] Additionally, the logic flows shown in the figures do not require the particular order shown, that is, the sequential order, to achieve the desired result. Additionally, other steps may be provided in the described flows, steps may be eliminated from the described flows, other components may be added to the described systems, or other components may be removed from the described systems. Accordingly, other embodiments are within the scope of the following claims.

[0098] In addition to the above, for both the case where the systems, programs, or functions described in this specification may enable the collection of user information (e.g., information regarding the user's social network, social actions, or activities, occupation, user preferences, or the user's current location) and the case where they may enable it, and also for whether to send content or communications from the server to the user, control can be provided to the user to enable the user to make a selection. Additionally, certain data can be processed in one or more ways such that information that can identify an individual is removed before it is stored or used. For example, the user's identification information can be processed so that information that can identify the individual user cannot be determined, or, if location information is obtained (such as at the city, zip code, or state level), the user's geographical location can be generalized so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how that information is used, and what information is provided to the user.

[0099] As described herein, specific features of the described embodiments have been illustrated, but numerous modifications, substitutions, changes, and equivalents will occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. They are presented by way of example and not limitation, and it should be understood that various changes in form and detail may be made. Any part of the apparatus and / or method described herein can be combined in any combination, except mutually exclusive combinations. The embodiments described herein can include various combinations and / or sub-combinations of the functions, components, and / or features of the different embodiments described.

Claims

1. A method executed by a computer, the method comprising: determining a context associated with the operation of a head-mounted wearable device; determining a content output state of the head-mounted wearable device based on the context and one or more situational safety factors associated with the context; outputting content by a display device of the head-mounted wearable device based on the content output state of the head-mounted wearable device, wherein outputting the content includes: outputting the content within a display area of the head-mounted wearable device in a first content output state; delaying the output of the content in a second content output state; A method executed by a computer, including the above.

2. Determining the context includes: acquiring data associated with the head-mounted wearable device, the data including at least one of: location data provided by the head-mounted wearable device; image data provided by an image sensor of the head-mounted wearable device; or location data and orientation data provided by a location and orientation sensor of the head-mounted wearable device; The method executed by a computer according to Claim 1.

3. Acquiring the data associated with the head-mounted wearable device includes acquiring at least one of location data, image data, location data, or orientation data from a mobile computing device paired with the head-mounted wearable device. The method executed by a computer according to Claim 2.

4. determining the applicability of the content to the context; outputting the content in response to a determination that the content is applicable to the context; delaying the output of the content in response to a determination that the content is not applicable to the context; The method executed by a computer according to any one of the preceding claims, further comprising the above.

5. ​ Outputting the content includes outputting the content within a specified content display area in the display area of the head-mounted wearable device, and outputting the content identifying a portion of the display area of the head-mounted wearable device as the specified content display area; displaying the content only within the specified content display area; designating the remaining portion of the display area of the head-mounted wearable device as a non-display area; limiting the display of content in the non-display area; A method executed by a computer according to any one of the preceding claims, including **Claim 6** Designating the remaining portion of the display area of the head-mounted wearable device as a non-display area includes designating a portion of the display area of the head-mounted wearable device corresponding to the one or more situation safety factors as a non-display area; limiting the display of content in the non-display area so as to maintain the user's field of view through the non-display area; A method executed by a computer according to claim 5, including **Claim 7** detecting a change in the context in which the head-mounted wearable device is operated; updating the content output state of the head-mounted wearable device based on the changed context; The method according to any one of the preceding claims, further comprising **Claim 8** modifying the manner in which the content is output based on the changed context and the updated content output state. A method executed by a computer according to claim 7, further comprising **Claim 9** Modifying the manner in which the content is output includes simplifying the appearance of the content in response to a detected change in the one or more situation safety factors; changing the display position of the content in response to a detected change in the one or more situation safety factors; temporarily stopping the output of the content in response to a detected change in the one or more situation safety factors; resuming the output of the content in response to a detected change in the one or more situation safety factors; A method executed by a computer according to claim 8, comprising at least one of the above.

10. The first content output state is such that the output of the content within the specified content display area maintains the situational safety based on the context in which the head-mounted wearable device is operated, and the one or more situational safety factors associated with the operation of the head-mounted wearable device. A method executed by a computer according to any one of the preceding claims, which is a state.

11. The second content output state is such that the delay of the content maintains the situational safety based on the context in which the head-mounted wearable device is operated, and the one or more situational safety factors associated with the operation of the head-mounted wearable device. A method executed by a computer according to any one of the preceding claims, which is a state.

12. A non-transitory computer-readable medium storing executable instructions, which, when executed by at least one processor of a computing device, cause the at least one processor to, Determine a context associated with the operation of the head-mounted wearable device; Determine the content output state of the head-mounted wearable device based on the context and one or more situational safety factors associated with the context; Output content based on the content output state of the head-mounted wearable device by a display device of the head-mounted wearable device. Outputting the content Output the content within the display area of the head-mounted wearable device in a first content output state; Delay the output of the content in a second content output state; A non-transitory computer-readable medium comprising:

13. The instructions cause the one or more processors to, Obtain data associated with the head-mounted wearable device, the data Location data provided by at least one of the head-mounted wearable device or a mobile computing device paired with the head-mounted wearable device image data provided by an image sensor of the head-mounted wearable device, or position data and orientation data provided by a position and orientation sensor of at least one of the head-mounted wearable device or the mobile computing device, including at least one of the non-transitory computer-readable medium according to claim 12. **Claim 14** The instructions further cause the one or more processors to determine applicability of the content to the context, output the content in response to a determination that the content is applicable to the context, delay output of the content in response to a determination that the content is not applicable to the context, and perform the non-transitory computer-readable medium according to claim 12 or 13. **Claim 15** The instructions cause the one or more processors to identify a portion of the display area of the head-mounted wearable device as a specified content display area, display the content only within the specified content display area, designate the remaining portion of the display area of the head-mounted wearable device as a non-display area, restrict display of content in the non-display area, and perform, the non-transitory computer-readable medium according to any one of claims 12 to 14. **Claim 16** The instructions cause the one or more processors to designate a portion of the display area of the head-mounted wearable device corresponding to the one or more situational safety factors as a non-display area, restrict display of content in the non-display area so as to maintain the user's field of view through the non-display area, and perform, the non-transitory computer-readable medium according to claim 15. **Claim 17** detect a change in the context in which the head-mounted wearable device is operated, update the content output state of the head-mounted wearable device based on the changed context, modify the manner in which the content is output based on the changed context and the updated content output state, and further include, the non-transitory computer-readable medium according to any one of claims 12 to 16. **Claim 18** The command causes the at least one processor to modify the format in which the content is output, the format being to simplify the appearance of the content in response to a detected change in the one or more situation safety factors, to change the display position of the content in response to a detected change in the one or more situation safety factors, to pause the output of the content in response to a detected change in the one or more situation safety factors, to resume the output of the content in response to a detected change in the one or more situation safety factors, The non-transitory computer-readable medium according to claim 17, comprising.

19. The first content output state is a state in which the output of the content within the display area maintains the situation safety based on the context in which the head-mounted wearable device is operated and the one or more situation safety factors associated with the operation of the head-mounted wearable device. The non-transitory computer-readable medium according to any one of claims 12 to 18.

20. The second content output state is a state in which the delay of the content maintains the situation safety based on the context in which the head-mounted wearable device is operated and the one or more situation safety factors associated with the operation of the head-mounted wearable device. The non-transitory computer-readable medium according to any one of claims 12 to 19.

Images