Content output management in head wearable devices

Abstract

A system and method for selectively outputting content on a head-mounted wearable computing device is provided. The system may determine a context associated with operation of the head-mounted wearable computing device, and selectively output the content on the head-mounted wearable computing device based on the context, or delay output of the content. Content may be displayed in one or more specified portions of a display of the head-mounted wearable computing device, thereby reducing distraction of a user and enhancing contextual awareness and contextual security during use of the head-mounted wearable computing device.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation of and claims priority to U.S. application No. 17 / 812,903, filed on July 15, 2022, entitled “CONTENT OUTPUT MANAGEMENT IN A HEAD MOUNTED WEARABLE DEVICE,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] The present specification generally relates to managing the output of content in a head-mounted wearable device, and specifically to improving situational safety based on selective output of content based on situational context in a head-mounted wearable device. Background Art

[0003] Wearable devices may include: head-mounted devices, including, for example, glasses including smart glasses, headphones, earbuds, etc.; wrist-mounted and / or hand-mounted devices, including, for example, watches including smart watches, smart bracelets, smart rings, etc.; smart pendants; fitness trackers; cameras; body sensors; and other such devices. In some scenarios, distractions caused by, for example, user interactions with displayed content, incoming notifications, etc. may compromise safety. In the case of smart glasses, the displayed content may not only cause distraction, but may also obstruct the user's visibility of the surrounding environment. In some scenarios, such as professional and / or social interactions, users may want to avoid these types of distractions in order to stay engaged in the interaction. Summary of the invention

[0004] In a first general aspect, a computer-implemented method includes: determining a context associated with operation of a head-mounted wearable device; determining a content output state of the head-mounted wearable device based on the context and one or more situational safety factors associated with the context; and outputting content by a display device of the head-mounted wearable device based on the content output state of the head-mounted wearable device. Outputting the content may include: outputting the content in a display area of ​​the head-mounted wearable device in a first content output state; and delaying output of the content in a second content output state.

[0005] In some implementations, determining the context includes obtaining data associated with the head-mounted wearable device, the data including at least one of: positioning data provided by the head-mounted wearable device, image data provided by an image sensor of the head-mounted wearable device, or position and orientation data provided by a position and orientation sensor of the head-mounted wearable device. Obtaining the data associated with the head-mounted wearable device may include obtaining at least one of the positioning data, image data, position data, or orientation data from a mobile computing device paired with the head-mounted wearable device.

[0006] In some implementations, the computer-implemented method may further include determining suitability of the content for the context; outputting the content in response to determining that the content is suitable for the context; and delaying output of the content in response to determining that the content is not suitable for the context.

[0007] In some implementations, outputting content includes outputting content in a designated content display area within a display area of ​​a head-mounted wearable device, including: identifying a portion of the display area of ​​the head-mounted wearable device as the designated content display area; displaying the content only in the designated content display area; and designating the remainder of the display area of ​​the head-mounted display device as a non-display area; and limiting display of the content in the non-display area. Designating the remainder of the display area of ​​the head-mounted wearable device as the non-display area may include: designating a portion of the display area of ​​the head-mounted wearable device corresponding to one or more situational safety factors as the non-display area; and limiting display of the content in the non-display area, thereby maintaining a user's field of view through the non-display area.

[0008] In some implementations, the computer-implemented method may further include: detecting a change in the context in which the head-mounted wearable device operates; and updating the content output state of the head-mounted wearable device based on the changed context. In some implementations, the computer-implemented method may further include modifying the manner in which the content is output based on the changed context and the updated content output state. Modifying the manner in which the content is output may include at least one of the following: simplifying the appearance of the content in response to a detected change in one or more situational safety factors; changing the display position of the content in response to a detected change in one or more situational safety factors; pausing the output of the content in response to a detected change in one or more situational safety factors; or resuming the output of the content in response to a detected change in one or more situational safety factors.

[0009] In some implementations, the first content output state is a state in which content output in a designated content display area maintains a context-safe state based on the context in which the head-mounted wearable device is operated and one or more context-safe factors associated with the operation of the head-mounted wearable device. In some implementations, the second content output state is a state in which a delay in content output maintains a context-safe state based on the context in which the head-mounted display is operated and one or more context-safe factors associated with the operation of the head-mounted wearable device.

[0010] In another general aspect, a non-transitory computer-readable medium stores executable instructions that, when executed by at least one processor of a computing device, cause the at least one processor to: determine a context associated with operation of a head-mounted wearable device; determine a content output state of the head-mounted wearable device based on the context and one or more situational safety factors associated with the context; and output content by a display device of the head-mounted wearable device based on the content output state of the head-mounted wearable device. Outputting the content may include: outputting the content in a display area of ​​the head-mounted wearable device in a first content output state; and delaying the output of the content in a second content output state.

[0011] In some implementations, the instructions cause at least one processor to obtain data associated with a head-mounted wearable device, the data comprising at least one of: positioning data provided by at least one of the head-mounted wearable device or a mobile computing device paired with the head-mounted wearable device; image data provided by an image sensor of the head-mounted wearable device; or position and orientation data provided by a position and orientation sensor of at least one of the head-mounted wearable device or the mobile computing device.

[0012] In some implementations, the instructions cause at least one processor to: determine suitability of the content for the context; output the content in response to determining that the content is suitable for the context; and delay output of the content in response to determining that the content is not suitable for the context.

[0013] In some implementations, the instructions cause at least one processor to: identify a portion of the display area of ​​the head-mounted wearable device as a designated content display area; display content only in the designated content display area; designate the remainder of the display area of ​​the head-mounted display device as a non-display area; and limit the display of content in the non-display area.

[0014] In some implementations, the instructions cause at least one processor to: designate a portion of a display area of ​​the head-mounted wearable device that corresponds to one or more situational safety factors as a non-display area; and limit display of content in the non-display area to maintain a user's field of view through the non-display area.

[0015] In some implementations, the instructions cause at least one processor to: detect a change in a context in which the head-mounted wearable device operates; update a content output state of the head-mounted wearable device based on the changed context; and modify a manner in which content is output based on the changed context and the updated content output state. In some examples, the instructions cause at least one processor to modify a manner in which content is output, including: simplifying the appearance of content in response to a detected change in one or more situational safety factors; changing a display position of content in response to a detected change in one or more situational safety factors; pausing output of content in response to a detected change in one or more situational safety factors; or resuming output of content in response to a detected change in one or more situational safety factors.

[0016] In some implementations, the first content output state is a state in which content output in the display area remains contextually safe based on the context in which the head-mounted wearable device is operating and one or more contextual safety factors associated with the operation of the head-mounted wearable device.

[0017] In some implementations, the second content output state is based on the context in which the head-mounted display is operating and one or more contextual safety factors associated with the operation of the head-mounted wearable device, and the delay in content output maintains a contextually safe state.

[0018] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a third-person view of the user's surroundings.

[0020] Figure 2A yes Figure 1 A front view of an example wearable device is shown, and Figure 2B is a rear view of the example wearable device.

[0021] Figure 2C yes Figure 1 A front view of an example mobile device is shown.

[0022] Figure 3 is a block diagram of a system according to implementations described herein.

[0023] FIG. 4A to FIG. 4D Operation of an example wearable device in a first example use environment is illustrated.

[0024] FIG. 5A to FIG. 5E Operation of an example wearable device in a second example use environment is shown.

[0025] Figure 6 is a flow chart of an example method according to implementations described herein.

[0026] Figure 7 An example computing device of the computing systems discussed herein is shown. DETAILED DESCRIPTION

[0027] The present disclosure relates to systems and methods for improving safety in the use of wearable devices. In some examples, the systems and methods disclosed herein can improve the safety of the use of wearable devices by reducing user distractions, particularly in scenarios where user distractions may be dangerous. In some examples, these systems and methods can selectively determine whether to output information to a user via a wearable device based on a determination of the context in which the wearable device operates. In some examples, these systems and methods can use the determination of the context to determine whether the wearable device is in a first content output state, in a second content output state, or in a third content output state, in which the content can be output when it becomes available, in the second content output state, only in a specified portion of the output device and / or in a modified manner and / or form, and in the third content output state, the output of the content is delayed or paused. In some examples, these systems and methods can make a determination of how and / or where to output information to the user based on a determination of the context in which the wearable device is used. The systems and methods according to the implementations described herein can adapt information output to a user based on the context in which the wearable device is used and the information to be output, thereby improving the safety of use of the wearable device.

[0028] Hereinafter, for the purpose of discussion and illustration only, the systems and methods according to the implementations described herein will be described with respect to the safety associated with the use of a wearable device in the form of a head-mounted wearable device (such as smart glasses with display capabilities and / or computing capabilities). The principles to be described herein may be applicable to other types of wearable devices, and in particular to other types of wearable devices where selective information output to a user based on the current context in which the wearable device is used can improve the safety of the use of the wearable device.

[0029] Figure 1is a third-person view of the user, including various mobile devices. The example mobile devices include various example wearable devices, including a first wearable device 100 in the form of a head-mounted wearable device or glasses such as smart glasses worn on the user's head, a second wearable device 180 in the form of earplugs worn in one or both ears of the user, and a third wearable device 190 in the form of a watch such as a smart watch worn on the user's wrist. Figure 1 A mobile device 200 in the form of a handheld computing device such as a smart phone held by a user is included. In some examples, the first wearable device 100 and / or the second wearable device 180 and / or the third wearable device 190 and / or the mobile device 200 may include output capabilities, including display output capabilities and / or audio output capabilities. In some examples, the first wearable device 100 and / or the second wearable device 180 and / or the third wearable device 190 and / or the mobile device 200 may include computing and / or processing capabilities. In some examples, the first wearable device 100 and / or the second wearable device 180 and / or the third wearable device 190 and / or the mobile device 200 may access additional computing and processing resources via a network and / or are locally available on the mobile device 200 and / or one of the wearable devices 100, 180, 190. In some examples, the wearable devices 100, 180, 190 and the mobile device 200 can communicate with each other and / or with external resources to exchange information, receive and send input and / or output, etc. The principles to be described herein can be applied to Figure 1 Other types of wearable devices and / or mobile devices not specifically shown.

[0030] Hereinafter, for the purpose of discussion and illustration, systems and methods will be described with respect to safety improvements associated with the operation of the first example wearable device 100 in different contextual scenarios, either alone or in combination with the example mobile device 200. The principles to be described herein may be applicable to achieving safety improvements associated with the operation of other types of wearable devices, either alone or in combination with the example mobile device 200 and / or other types of mobile devices.

[0031] For discussion and illustration purposes, Figure 2A and Figure 2B An example head-mounted wearable device 100 is shown in the form of a pair of smart glasses. In particular, Figure 2A is a front view of an example head mounted wearable device 100, and Figure 2B1 is a rear view of the example head-mounted wearable device. The example head-mounted wearable device 100 includes a frame 102 having a rim portion 103 surrounding a glass portion 107 or a lens 107 and an arm portion 105 coupled to the corresponding rim portion 103. In some examples, the lens 107 can be a corrective / prescription lens. In some examples, the lens 107 can be a glass portion that does not necessarily include a corrective / prescription parameter. A nose bridge portion 109 can connect the rim portion 103 of the frame 102.

[0032] Figure 2A and Figure 2B The example wearable device 100 shown includes a display device 104 that can output visual content, for example, at an output coupler 144, so that the visual content can be visible to the user. In some examples, the output coupler 144 can be substantially coincident with the lens 107. In this form, the wearable device 100 can also include an audio output device 106 (such as, for example, one or more speakers), an illumination device 108, a sensing system 110, a control system 112, at least one processor 114, and an outward-facing image sensor 116 or camera 116.

[0033] In some examples, the display device 104 may include a see-through near-eye display. For example, the display device 104 may be configured to project light from a display source onto a portion of a teleprompter glass that acts as a beam splitter that sits at an angle (e.g., 30 to 45 degrees). The beam splitter may implement reflection and transmission values ​​that allow light from the display source to be partially reflected and the remaining light to be transmitted. This optical design may allow a user to simultaneously see, for example, physical items in the world through the lens 107, alongside content (e.g., digital images, user interface elements, virtual content, etc.) generated by the display device 104. In some implementations, a waveguide, holographic panel, or other combiner optical device may be used to depict content from the display device 104.

[0034] In some examples, the gaze tracking device 120 can detect and track eye gaze direction and movement. The data captured by the gaze tracking device 120 can be processed to detect and track gaze direction and movement as user input. In some implementations, the sensing system 110 can include various sensing devices, and the control system 112 can include various control system devices, including, for example, one or more processors 114 operably coupled to components of the control system 112. In some implementations, the control system 112 can include a communication module that provides communication and information exchange between the wearable device 100 and other external devices.

[0035] The example wearable device 100 may include more or fewer features than those described above. The principles to be described herein are applicable to head-mounted wearable devices with or without corrective lenses.

[0036] Figure 2C yes Figure 1 2. A front view of an example mobile device 200 is shown. The example mobile device 200 may include an interface device 210. In some implementations, the interface device 210 may function as an input device, including, for example, a touch surface 212 that may receive touch input from a user. In some implementations, the interface device 210 may function as an output device, including, for example, a display portion 214 that allows the interface device 210 to output information to a user. In some implementations, the interface device 210 may function as both an input device and an output device. The example mobile device 200 may include an audio output device 216 or speaker that outputs audio signals to a user.

[0037] The example mobile device 200 may include a sensing system 220 that includes various sensing system devices. In some examples, the sensing system devices include, for example, one or more image sensors, one or more position and / or orientation sensors, one or more audio sensors, one or more touch input sensors, and other such sensors. Figure 2C The example mobile device 200 shown includes an image sensor 222 that is included in a forward-facing camera of the mobile device 200. The example mobile device 200 may include additional image sensors, such as, for example, a world-facing camera. Figure 2C The example mobile device 200 shown includes an inertial measurement unit (IMU) 224, which includes, for example, position and / or orientation and / or acceleration sensors, such as, for example, accelerometers, gyroscopes, magnetometers, and other such sensors that can provide position and / or orientation and / or acceleration data. Figure 2C The illustrated example mobile device 200 includes an audio sensor 226 that can detect audio signals, for example, to be processed as user input. Figure 2CThe example mobile device 200 shown includes, for example, a touch input sensor 228 corresponding to the touch surface 212 of the interface device 210. The touch input sensor 228 can detect touch input signals for processing as user input. The example mobile device 200 may include a control system 270, which includes various control system devices. The example mobile device 200 may include a processor 290 for facilitating the operation of the mobile device 200. The example mobile device 200 may include a communication module 250 that provides connection to a network, communication with other devices external to the example mobile device 200 (such as, for example, the example wearable device 100 and other such devices), etc. In some examples, access to a network can be shared with external devices such as the example wearable device 100 via the example mobile device 100.

[0038] Systems and methods according to implementations described herein may provide for determination of contextual information associated with current use of a wearable device. In some examples, contextual information may be considered when determining whether to output content to a user or whether to delay output of content to a user. In some examples, contextual information may be considered when determining a manner in which content is output to a user. In the case where the wearable device is a head-mounted wearable device (such as a wearable head-mounted device), contextual information may be considered when determining whether to output content to a user. Figure 1 , Figure 2A and Figure 2B In the example of the wearable device 100 shown in FIG. 1 , contextual information can be used to determine, for example, whether the output of content may be distracting to the user, whether the output of content is relevant or applicable to the user at a particular time based on the contextual information, whether the output of content can be adapted so that the output of content minimizes distraction from the user's situational awareness or avoids obstructing objects in front of the user, etc. The systems and methods according to the implementations described herein provide selective output of content through the wearable device 100 in a manner that helps the user maintain situational awareness and situational safety.

[0039] Figure 3 is a block diagram of an example system for performing the systems and methods described herein. The system may include one or more computing devices. Figure 1 , Figure 2A and Figure 2B An example computing device 300 in the form of a head mounted wearable device 100 is included in Figure 3 In the example system shown in FIG. 1 , computing device 300 may operate under the control of control system 340. Computing device 300 may communicate directly (via wired and / or wireless communications) or via network 305 with one or more external devices 370 (such as, for example, another wearable computing device, such as a wearable computing device, a wearable computing device, or a wearable computing device such as a wearable computing device). Figure 1 and Figure 2CThe computing device 300 may communicate with another mobile computing device (such as another mobile computing device of the mobile device 200 shown in the figure). In some implementations, the computing device 300 includes a communication module 350 to facilitate external communication. In some implementations, the computing device 300 includes a sensing system 310, which includes various sensing system components, including, for example, one or more image sensors 312, one or more position / orientation sensors 314 (including, for example, inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.), one or more audio sensors 316 that can detect audio input, one or more touch input sensors 318 that can detect touch input, and other such sensors. The computing device 300 may include more or fewer sensing devices and / or combinations of sensing devices.

[0040] In some examples, image sensor 312 may include, for example, cameras, such as, for example, outward-facing or world-facing cameras, etc., which can capture still and / or moving images of the environment outside computing device 300. Still and / or moving images can be displayed by a display device of output system 330, and / or sent to the outside via communication module 350 and network 305, and / or stored in memory 320 of computing device 300. Computing device 300 may include one or more processors 360. Processor 360 may include various modules or engines configured to perform various functions. In some examples, processor 360 may include an object classification module, a pattern classification module, a configuration identification module, and other such processors. Processor 360 may be formed in a substrate that is configured to execute one or more machine executable instructions or one or more pieces of software, firmware, or a combination thereof. Processor 360 may be semiconductor-based, including semiconductor materials that can execute digital logic. Memory 320 may include any type of storage device that stores information in a format that can be read and / or executed by processor 360. Memory 320 may store applications and modules that perform certain operations when executed by processor 360. In some examples, these applications and modules may be stored in an external storage device and loaded into memory 320.

[0041] The computing device 300 can access additional resources 390 to facilitate the determination of a number of factors to be considered when selectively outputting content to a user via the computing device 300. In some examples, the additional resources 390 are available locally on the computing device 300. In some examples, the additional resources 390 are available to the computing device 300 via the network 305. In some examples, the additional resources 390 are available to the computing device via an external device 370. In some examples, some of the additional resources 390 are available locally on the computing device 300, and some of the additional resources are available via the network 305 and / or via the external device 370. The additional resources 390 may include, for example, a server computer system, a processor, a database, a memory storage, etc. In some examples, the additional resources 390 may include various processing engines and / or modules, models (including machine learning models), algorithms, etc. In some examples, the additional resources 390 may include an object and / or pattern classification engine. In some examples, the object and / or pattern classification engine may detect and / or identify one or more features in the image data sent to it for processing. Additional resources 390 may include one or more models and / or one or more algorithms. In some examples, the models and / or algorithms may be executed by a processor to determine context information associated with the use of one or more computing devices included in the system, application of rules for information output, form and / or manner of outputting information, changes in the form and / or manner of outputting information, and other such processes.

[0042] As described above, in some examples, external device 370 can communicate with computing device 300. For example, external device 370 can be paired or operatively coupled with computing device 300. In some examples, external device 370 can provide processing and / or computing resources to computing device 300. In some examples, the external device can provide additional data used in the determination of context and the selective output of content to a user of computing device 300. As described above, in some examples, external device 370 can be in the form of, for example, Figure 1 and Figure 2CIn some examples, the external device 370 can operate under the control of the control system 380. The external device 370 can communicate with one or more other devices directly (via wired and / or wireless communication) or via the network 305. The external device can include a communication module 382 to facilitate external communication. The external device 370 can include a sensing system 371, which includes various sensing system components, including, for example, one or more image sensors 372, one or more position / orientation sensors 373 (including, for example, inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.), one or more audio sensors 374 that can detect audio input, one or more touch input sensors 375 that can detect touch input, and other such sensors.

[0043] The external device may include an output system 377, which includes one or more display devices that can output image content and one or more audio output devices that can output audio content. The external device 370 may include one or more processors 384. The processor 384 may include various modules or engines configured to perform various functions. The processor 384 may be formed in a substrate that is configured to execute one or more machine executable instructions or one or more pieces of software, firmware, or a combination thereof. The processor 384 may be semiconductor-based, including semiconductor materials that can execute digital logic. The memory 376 may include any type of storage device that stores information in a format that can be read and / or executed by the processor 384. The memory 376 may store applications and modules that perform certain operations when executed by the processor 384. In some examples, these applications and modules may be stored in an external storage device and loaded into the memory 376.

[0044] In the following, we will discuss Figure 1 , Figure 2A and Figure 2B The systems and methods are described with reference to a wearable device in the form of an example head-mounted wearable device 100 shown in FIG. In some examples, the operation of the example head-mounted wearable device 100 is described alone or in combination with operations such as Figure 1 and Figure 2C The systems and methods are described with reference to an external device such as the example mobile device 200. The principles to be described herein may be applicable to other types of wearable devices and / or combinations of devices.

[0045] In some examples, a user wearing an example head-mounted wearable device 100 can consume content output by the head-mounted wearable device 100. In some examples, the content can be output in the form of image content output by the display device 104 and visible within the user's field of view. In some examples, the content can be in the form of audio content output by the audio output device 106. In some examples, the content can include a combination of image content and audio content. In some scenarios, the output of content by the head-mounted wearable device 100 may distract the user or block important objects in front of the user. In some scenarios, such distraction or blockage may endanger the safety of the user and / or other people near the user, or cause danger to the user and / or those near the user. These types of scenarios may require delaying the output of information to the user, modifying how information is output to the user, etc., to maintain situational safety when the head-mounted wearable device 100 is in use.

[0046] For example, when the user is participating in an activity that would otherwise benefit from the user's substantially undivided attention, the content and / or notifications displayed by the display device 104 may be a distraction to the user. In an example where the user is walking, for example, in a somewhat crowded environment (i.e., pedestrian traffic, obstacles in the user's path, car traffic, etc.), the content and / or notifications may cause user distraction, which may endanger the safety of the user and / or those near the user, particularly when the content and / or notifications are not relevant to or applicable to the current activity. Similarly, in an example where the user is driving, the display of content and / or notifications may cause distraction, which may endanger the safety of the user and / or those near the user. In addition, when the user is participating in activities such as walking or driving, the display of content and / or notifications may obstruct portions of the user's field of view in some scenarios, thereby creating a potential safety hazard. Many other types of environmental conditions that would benefit from reduced user distraction may similarly require delaying the output of information to the user, modifying how information is output to the user, etc. Such environmental conditions may include, for example, icy sidewalks and / or roads, precipitation that compromises visibility, and other such conditions.

[0047] Systems and methods according to the implementations described herein may determine a context associated with the use of a wearable device, and make a determination as to whether to output information or delay information output via a mobile device based on the determined context. When determining output information, systems and methods according to the implementations described herein may determine the manner in which the information is output based on the context associated with the use of the wearable device. For image-based or visual output, the manner in which the information is output may include, for example, determining the visual area of ​​the output information to minimize distraction and avoid obstructing important objects in the user's field of view. In some examples, this may include modifying the content to be output, such as merging and / or simplifying the information to be output to reduce the size and / or complexity level of the information to be output. In some examples, this may include delaying the output of the information until the context in which the wearable device is used has changed, and the output of the information no longer causes distraction or obstruction that may cause safety issues to the user.

[0048] Figure 4A is the third-person view of the user in the surrounding environment 400. Figure 4A In the example shown, the user wears the head-mounted wearable device 100 ( Figure 4A Handheld mobile computing device 200 ( Figure 4A 100 and the mobile device 200 may be carried by the user. In some examples, the wearable device 100 and the mobile device 200 may be paired or operably coupled. When in this state, the wearable device 100 and the mobile device 200 may communicate with each other, exchange data and information, share computing resources, etc. In some examples, the wearable device 100 and the mobile device 200 may operate substantially independently. FIG. 4B to FIG. 4D is a third-person view of the user's field of view of the surrounding environment 400 as viewed through one of the lenses 107 of the example wearable device 100. In particular, FIG. 4B to FIG. 4D 4 shows a user's field of view 405 of the surrounding environment 400 through one of the lenses 107 of the head-mounted wearable device 100. In this example, the user's field of view can represent what is visible to the user when the user is viewing the surrounding environment 400, for example, through the lens 107 of the head-mounted wearable device 100. This user's field of view changes as, for example, the user changes / shifts head position, moves, etc., resulting in corresponding changes in what is visible to the user. In this example, FIG. 4B to FIG. 4D The user's field of view 405 through one of the lenses 107 of the head-mounted wearable device 100 is shown as the user walks along the sidewalk 410 toward the crosswalk 420 and then turns at the crosswalk 420 to cross the road.

[0049] exist Figure 4BIn the user's field of view 405 shown, the user is walking along the sidewalk 410 toward the crosswalk 420. Figure 4B In FIG. 4 , content in the form of a virtual object 450 output by the display device 104 is displayed to the user to provide route guidance. Figure 4B In the example shown, virtual object 450 includes relatively detailed route guidance, including text instructions and an estimated time of arrival, street, direction indicators, etc. In this example, virtual object 450 including relatively detailed route guidance may have been displayed in response to determining that the user is in a situation where the output of virtual object 450 including detailed route guidance and any distractions associated with the output of virtual object 450 are unlikely to compromise situational safety. This determination may be based on a variety of different situational safety factors and / or combinations of situational safety factors. These factors may include, for example, detecting that there is little to no pedestrian traffic on sidewalk 410, no upcoming turns in the route guidance provided by virtual object 450, no automobile traffic detected in the user's upcoming path, a relatively unobstructed path / no obstacles that could pose a hazard detected in the user's path, and other such factors. Figure 4B The example shown represents a content output state of the wearable device 100, in which the environmental conditions in which the wearable device 100 operates allow content to be output in a relatively relaxed manner.

[0050] exist Figure 4C In the user's field of view 405 shown, the user's position on the sidewalk 410 is Figure 4B The positions in are basically the same. However, in Figure 4C In , one or more obstacles or obstructions are detected in the user's path. Figure 4C In the example shown, example obstacles include pedestrian traffic on the sidewalk in the user's path, and trash cans on the sidewalk in the user's path. In this scenario, a virtual object 451 is output by the display device 104. Figure 4C The virtual object 451 shown includes modified or simplified route guidance. Figure 4C The virtual object 451 is shown to be somewhat transparent, so that the user's view 405 of the surrounding environment 400 is still somewhat visible to the user through the virtual object 451. The modified or simplified route guidance provided by the virtual object 451 can reduce distraction caused by the display of the virtual object 451, thereby allowing the user to maintain situational awareness and avoid obstacles along the path, while still maintaining awareness of the path to be followed according to the route guidance. This can improve situational safety during use of the wearable device 100. Figure 4CThe examples shown represent content output states of the wearable device 100 in which the environmental conditions in which the wearable device 100 operates require changes and / or additional control of the manner in which the content is output.

[0051] In some examples, the system can identify one or more areas or zones of the surrounding environment 400 within the user's field of view 405. Based on the context, obstacles, hazards, and other items to consider associated with the use of the wearable device 100, the system can identify one or more areas or zones within the user's field of view 405 where output of information, such as in the form of virtual objects 451, can be displayed. Figure 4C 4. The example content display areas 480 may correspond to areas or portions of the user's field of view 405 in which the display of virtual objects 451 is less likely to distract the user of the wearable device 100 and / or obstruct or block the user's view of safety-critical areas, thereby allowing the user to maintain situational awareness / situational safety while using the wearable device 100.

[0052] exist Figure 4C In the user's field of view 405 shown, the user's position on the sidewalk 410 is Figure 4B The positions in are basically the same. However, in Figure 4C In , one or more obstacles or obstructions are detected in the user's path. Figure 4C In the example shown, example obstacles include pedestrian traffic on the sidewalk in the user's path, and trash cans on the sidewalk in the user's path. In this scenario, a virtual object 451 is output by the display device 104. Figure 4C The virtual object 451 shown includes modified or simplified route guidance. Figure 4C The virtual object 451 is shown to be somewhat transparent, so that the user's view 405 of the surrounding environment 400 is still somewhat visible to the user through the virtual object 451. The modified or simplified route guidance provided by the virtual object 451 can reduce distraction caused by the display of the virtual object 451, thereby allowing the user to maintain situational awareness and avoid obstacles along the path, while still maintaining awareness of the path to be followed according to the route guidance. This can improve situational safety during use of the wearable device 100.

[0053] As described above, in some examples, the determination of whether to output information and / or in what manner and / or at what location within the user's field of view can be determined based on situational safety factors associated with the use of the wearable device 100. In some examples, these types of factors can be detected in image data captured, for example, by an image sensor 116 of the head-mounted wearable device 100. The image data can include a portion of the surrounding environment 400 in the user's path, an area around the user, etc., which can provide a representation of the situational safety factors to be considered. The image data can be generated by, for example, a classification engine (such as that included above with respect to Figure 3 The objects and / or patterns detected in the image data may be processed by, for example, one or more models, algorithms, etc., to determine the context in which the wearable device 100 is currently operating. In some examples, the determination of the context may include, for example, the determination of the positioning. This may include processing of positioning data provided by the wearable device 100 and / or by a mobile device 200 paired with the wearable device 100. In some examples, the determination of the context may include processing of position and / or orientation data provided by corresponding sensors of the wearable device 100 and / or the mobile device 200. In some examples, the position and / or orientation data may represent user movements corresponding to, for example, checking traffic conditions, changing directions, starting / stopping movement, using rearview mirrors or side mirrors, etc.

[0054] In some examples, the processing of detected objects and / or patterns and / or positioning can provide an indication of the environment in which the wearable device 100 is currently operating, the activity in which the user is engaged, etc., to determine the context. In some examples, the processing of detected objects and / or patterns and / or positioning can identify potential dangers near the user. In some examples, the processing of detected objects and / or patterns and / or positioning and corresponding context can be used to determine the placement of information output in the user's field of view 405. In some examples, the processing of detected objects and / or patterns and / or positioning and corresponding context can be used to determine the area within the user's field of view 405 where information (e.g., in the form of virtual objects) can be displayed to the user without causing danger. In some examples, the processing of detected objects and / or patterns and / or positioning and corresponding context can be used to determine the area within the user's field of view 405 where the display of information (e.g., in the form of virtual objects) may cause danger to the user and / or those near the user.

[0055] exist Figure 4D In the user's view 405 shown, the user has reached the corner of the sidewalk 410 and has turned toward the crosswalk 420 in preparation for crossing the street. Figure 4D, the system has detected that the user has reached crosswalk 420. The system can detect the user's position at crosswalk 420 preparing to cross the street, and can pause the display of information (e.g., in the form of virtual objects 450, 451). The user's position at crosswalk 420 can be detected based on processing and analysis of image data obtained, for example, by image sensor 116 of wearable device 100, positioning data provided by wearable device 100 and / or mobile device 200, and the like. For example, a pattern associated with crosswalk 420 can be detected in the image data. One or more cars can be detected in the image data. Positioning data including, for example, global positioning system data, traffic data, position and / or orientation data provided by corresponding sensors of wearable device 100 and / or mobile device 200, and the like can provide information related to the level of danger associated with walking and / or driving in a particular area. Figure 4D In the example shown, the system has paused the display of information at this point to eliminate distractions in a situation that is determined to benefit from the user's substantially undivided attention. That is, the system has paused the display of information to allow the user to focus on environmental / situational safety issues such as oncoming vehicular traffic, pedestrian traffic, traffic signals, etc. when crossing a street. In some examples, the system can resume the output of information / display of virtual objects (such as route guidance provided by example virtual objects 450 and 451) when it is determined that situational safety considerations require resuming the output of information. Figure 4D The examples shown represent content output states of the wearable device 100 in which the environmental conditions in which the wearable device 100 operates require additional changes and / or additional control of the manner in which the content is output (as opposed to the environment in which the wearable device 100 operates). Figure 4C Compared with the content output state shown, that is, compared with the content output state in which the wearable device 100 does not output content or temporarily pauses or delays content output).

[0056] Figure 4C It is shown that the system can adapt or modify the display of information based on the context and surrounding environment in which the wearable device 100 is used. Figure 4D The system is shown to delay or temporarily suspend the output of information based on the context and surroundings in which the wearable device 100 is used. In some examples, the system may allow the output of some types of information and delay the output of other types of information. The determination of whether to allow the output of information or delay the output of information may be associated with the context and surroundings in which the wearable device 100 is used and the relevance or applicability of the information to be output relative to the context and surroundings. FIG. 4A to FIG. 4DIn the example scenario shown, the system may determine that information in the form of route guidance is relevant or applicable to the current use of the wearable device 100, and may output the information (e.g., in the form of virtual objects 450, 451) when it is determined that the output of the information does not create an excessive distraction and pose a safety hazard. FIG. 4A to FIG. 4D In the example scenario shown, the system may determine that another type of information (such as, for example, a notification of an incoming message) may not be relevant or applicable to the current use of the mobile device 100 and / or may create an unnecessary distraction when received. In this scenario, the system may delay the output of the information (i.e., the display of the notification of the incoming message) until it is determined that the output of the information will not cause excessive distraction and / or compromise the safety of the situation.

[0057] FIG. 5A to FIG. 5E is a third person view of the user's field of view of the example wearable device 100. In particular, FIG. 5A to FIG. 5E 100 is a third-person view of a user's field of view 505 of a surrounding environment 500 through one of the lenses 107 of a head-mounted wearable device 100 worn by the user. In this example, the user's field of view may represent what is visible to the user when the user is viewing the surrounding environment 500, for example, through the lens 107 of the head-mounted wearable device 100. This field of view changes as, for example, the user changes / shifts head position, moves, etc., resulting in corresponding changes to what is visible to the user. In this example, FIG. 5A to FIG. 5E 5 shows a user's field of view 505 through one of the lenses 107 of the head mounted wearable device 100 while the user is sitting in the driver's seat of a vehicle. In this example, the handheld mobile computing device 200 ( FIG. 5A to FIG. 5E The wearable device 100 and the mobile device 200 may be in a storage location with the user, for example, in a vehicle interior. In some examples, the wearable device 100 and the mobile device 200 may be paired or operably coupled. When in this state, the wearable device 100 and the mobile device 200 may communicate with each other, exchange data and information, share computing resources, etc. In some examples, the wearable device 100 and the mobile device 200 may operate substantially independently.

[0058] In some examples, the context associated with the use of the wearable device 100 can be determined based on, for example, images captured by the image sensor 116 of the wearable device 100. In some examples, the context associated with the use of the wearable device 100 can be determined based on positioning data provided by the wearable device and / or the mobile device 200. In some examples, the context associated with the use of the wearable device 100 can be determined based on location and / or orientation data provided by sensors of the wearable device 100 and / or the mobile device 200. In some examples, the determination of whether to output information and / or in what manner to output information and / or at what location within the user's field of view to output information can be determined based on situational safety factors associated with the use of the wearable device 100. In some examples, these types of factors can be detected in image data captured by the image sensor 116 of the head-mounted wearable device 100, which can provide a representation of the situational safety factors to be considered. The image data can be detected by, for example, a recognition engine (such as that included above with respect to Figure 3 The objects and / or patterns detected in the image data may be processed by, for example, one or more models, algorithms, etc., to determine the context in which the wearable device 100 is currently operating. In some examples, the determination of the context may include processing of positioning data provided by the wearable device 100 and / or by the mobile device 200. In some examples, the determination of the context may include processing of position and / or orientation data provided by corresponding sensors of the wearable device 100 and / or the mobile device 200, which may represent user movements corresponding to, for example, checking traffic conditions, changing directions, starting / stopping movement, etc.

[0059] For example, processing of image data obtained by the image sensor 116 of the wearable device 100 can detect objects such as a steering wheel 510, a rearview mirror 520, a dashboard 530, an instrument cluster 540, a boundary 550 of a windshield 555, and other such objects. The positions and / or relative positions of these detected objects in the image data can provide an indication that the user of the wearable device is in the driver's seat of the vehicle. Similarly, objects such as, for example, a boundary 560 of a road 565, a dividing line 562, etc. can be detected. Processing of positioning information provided by the wearable device 100 and / or the mobile device 200 can indicate that the vehicle is in motion, and information related to road and / or traffic conditions, upcoming hazards, etc. can be provided. Processing of the position and / or orientation sensors of the wearable device 100 and / or the mobile device 200 can provide an indication of changes in the user's position and / or orientation and / or gaze within the vehicle. This data can be used to develop contextual understanding of the environment in which the wearable device 100 operates. Thus, objects and / or patterns detected in the image data, and / or positioning data, and / or location / orientation data provided by sensors of the wearable device 100 and / or the mobile device 200 may provide an indication of the environment in which the wearable device 100 is currently operating, the activity in which the user is engaged, etc., to determine the context. FIG. 5A to FIG. 5E In the arrangement shown, based on analysis of the data, it can be determined that the user of the wearable device 100 is in the driver's seat of the vehicle and is the operator of the vehicle.

[0060] In some examples, image data and / or positioning data and / or position / orientation data may be collected substantially continuously and processed substantially continuously, particularly when the vehicle is moving, to identify potential dangers near the user. In some examples, the processing of detected objects and / or patterns and / or positioning and / or position / orientation and corresponding context may be used to determine the placement of information output in the user's field of view 505. In some examples, the processing of detected objects and / or patterns and / or positioning and / or position / orientation and corresponding context may be used to determine areas within the user's field of view 505 where information (e.g., in the form of virtual objects) may be displayed to the user without causing excessive levels of distraction and without causing danger. In some examples, the processing of detected objects and / or patterns and / or positioning and / or position / orientation data and corresponding context may be used to determine areas within the user's field of view 505 where the display of information (e.g., in the form of virtual objects) may cause danger to the user and / or those near the user. For example, processing of detected objects and / or patterns and / or positioning and / or location / orientation data and corresponding context may be used to determine areas within the user's field of view that do not obstruct portions of the user's field of view 505 that allow continued safe use of the wearable device 100. FIG. 5A to FIG. 5EIn the driving example shown, this type of data and analysis can provide a determination of areas within the user's field of view 505 where information can be displayed, while allowing the driver / user of the wearable device 100 to maintain situational awareness and situational safety.

[0061] Figure 5B An example content display area 580 is shown. The example content display area 580 can define an area, zone, or portion of the user's field of view 505 in which the display of information is less likely to cause distraction to the user of the wearable device 100. The example content display area 580 can correspond to an area, zone, or portion of the user's field of view 505 in which the output of information (e.g., in the form of a display of virtual objects) does not obstruct or impede the user's view of safety-critical areas, thereby allowing the user to maintain situational awareness and situational safety while using the wearable device 100. For purposes of discussion and illustration, the example content display area 580 is shown in FIG. Figure 5B An example content display area 580 is shown in FIG. Figure 5C Output of information in the form of a virtual object 570 in a designated content display area 580 is shown.

[0062] exist Figure 5B and Figure 5C In the example arrangement shown, the content display area 580 corresponds to a portion of the instrument panel 530. For example, the output of information in the form of a display of a virtual object 570 in the designated content display area 580 will allow the user's field of view through the windshield 555 (including the peripheral portion of the windshield 555) to remain clear and unobstructed by the output of the information. Similarly, the display of the virtual object 570 in the designated content display area 580 allows the user's field of view of the rearview mirror 520 and the user's field of view of the instrument cluster 540 to remain clear and unobstructed by the display of the virtual object 570. In some examples, one or more non-content display areas may be designated within the user's field of view 505. The non-content display area may correspond to a portion of the user's field of view 505 that is to remain clear and unobstructed. Thus, in Figure 5B In the example shown, a portion of the user's field of view 505 corresponding to the windshield 555 can be designated as a first non-content display area 559. A portion of the user's field of view 505 corresponding to the rearview mirror 520 can be designated as a second non-content display area 529. A portion of the user's field of view 505 corresponding to the instrument cluster 540 can be designated as a third non-content display area 549.

[0063] For example, the output of information in the form of a virtual object 570 in the designated content display area 580 can reduce user distraction caused by the information output. The output of information in the designated content display area 580 can allow the remainder of the user's field of view 505 (i.e., the non-content display areas 529, 549, 559) to remain clear and visible to the user of the wearable device 100. This can allow the user to maintain situational awareness and situational safety when using the wearable device 100 in such a scenario.

[0064] exist Figure 5C In the example shown, content in the form of a virtual object 570 output by the display device 104 is displayed to the user in a content display area 580 of the user's field of view 505. In this example, the content in the form of the virtual object 570 provides route guidance. Figure 5C In the example shown, the virtual object 570 includes relatively detailed route guidance, including text instructions, exit numbers, estimated arrival times, direction indicators, and the like. In this example, the virtual object 570 including relatively detailed route guidance may have been displayed in response to a determination that the user is in a situation where the output of the virtual object 570 including detailed route guidance and any distractions associated with the output of the virtual object 570 are unlikely to compromise situational safety. This determination may be based on a variety of different situational safety factors and / or combinations of situational safety factors. In this particular example, these factors may include, for example, detection of little other vehicular traffic in the vicinity of the user, a relatively straight driving path / route ahead, no upcoming turns / exits in the route guidance provided by the virtual object 570, no upcoming stops, obstacles, hazards, and / or other such factors. In some examples, these factors may be determined based on, for example, positioning data provided by the wearable device 100 and / or a mobile device 200 co-located and communicating with the wearable device 100, image data obtained by the image sensor 116 of the wearable device 100, and other such data. FIG. 5A to FIG. 5C In the example shown, based on the context and environmental conditions in which the wearable device 100 operates, the content output state of the wearable device 100 determines the controlled output of content that is only relevant or applicable to the current context (i.e., driving) and only in the designated content display area 580.

[0065] exist Figure 5D In the user's field of view 505 shown, the user's position on the road is Figure 5C However, in Figure 5D In , one or more obstacles or obstructions are detected in the user's path. That is, in Figure 5DIn the example shown, the road has become more busy, potentially requiring the user to focus and pay attention. In this scenario, the virtual object 571 is output by the display device 104 and displayed in the content display area 580. Figure 5D The virtual object 571 shown includes modified or simplified route guidance. Figure 5D The virtual object 571 is shown to be somewhat transparent, such that the user's view 505 of the surrounding environment 500 is still somewhat visible to the user through the virtual object 571. The modified or simplified route guidance provided by the virtual object 571 can reduce distraction caused by the display of the virtual object 571, thereby allowing the user to maintain situational awareness as the user navigates through traffic while still maintaining awareness of the route to be followed according to the route guidance. In this example scenario, this can improve situational safety during use of the wearable device 100. Figure 5D In the example shown, the content output state of the wearable device 100 may be based on changes in context and changes in the environmental conditions in which the wearable device 100 operates, thereby requiring additional changes and / or additional control of the manner in which the content is output (as with Figure 5C compared to the content output status shown).

[0066] As described above, in some examples, the determination of whether to output information and / or the manner in which to output information and / or the location within the user's field of view to output information may be determined based on the determination of the usage context of the wearable device 100 and the situational safety factors associated with the use of the wearable device 100 in the determined context. FIG. 5A to FIG. 5E In the described examples, positioning data (e.g., GPS data, associated traffic data, route selection, etc.) accessible via the wearable device 100 and / or by a mobile device 200 paired with the wearable device 100 may provide at least some of the data used by the model and / or algorithm in making determinations regarding output of information. In some examples, the determination of context and whether to output information may include processing of position and / or orientation data provided by corresponding sensors of the wearable device 100 and / or the mobile device 200. In some examples, the position and / or orientation data may represent user movements corresponding to, for example, checking traffic conditions, changing direction, starting / stopping movement, etc. In some examples, these types of factors may be detected in image data captured, for example, by an image sensor 116 of the head-mounted wearable device 100 that captures an area surrounding the user. Positioning data and / or position / orientation data and / or image data may be processed by, for example, a model and / or algorithm (e.g., via the above description of the context). Figure 3 Additional resources described in 390 are available).

[0067] exist Figure 5EIn the user's field of view 505 shown, the user (temporarily) turns his / her head from the windshield 555 to the vehicle's side windows and side mirrors 525. Such movement may indicate, for example, that the user is performing a visual check of a blind spot, a visual check before changing lanes, etc. Figure 5E In , the system has detected such movement and has determined that the user's attention may be (temporarily) directed to a specific task, and the display of information (e.g., in the form of virtual objects 570, 571) can be suspended. That is, in order to provide the user with concentration on the current task (i.e., checking blind spots, changing lanes in this example), the system (temporarily) suspends the output of information (i.e., the display of virtual objects 570, 571). This can reduce distraction when the user is performing the current task, and can ensure that the user's field of view is not blocked or obstructed by the output of information when performing the task. Figure 5E In the example shown, the content output state of the wearable device 100 may be based on changes in context and changes in the environmental conditions in which the wearable device 100 operates, thereby requiring additional changes and / or additional control of the manner in which the content is output (as with Figure 5D compared to the content output status shown).

[0068] In some examples, a determination of this type of shift in focus for a user can be determined based on data provided by a position and / or orientation sensor of wearable device 100 and / or a position and / or orientation sensor of mobile device 200 and / or image data provided by image sensor 116 of wearable device 100. In some examples, a determination of a shift in focus (based on a change in head orientation) can be made based on data provided by a position and / or orientation sensor of wearable device 100, as compared to position / orientation data associated with mobile device 200. In this example, the position / orientation data associated with mobile device 200 can correspond to the position / orientation of the vehicle and, therefore, define a baseline position / orientation. The difference in position / orientation between the data from wearable device 100 and this baseline can confirm (in this example) that the user has shifted focus in a different direction. Figure 5EThe user may have turned his / her head in the manner shown. This comparison to the baseline movement provided by the mobile device 200 may allow the system to distinguish between an actual, intentional turn of the user's head and a change in the position and / or orientation of the user's head due to, for example, a turn of the vehicle (where a change in the position / orientation of the wearable device 100 would correspond to a change in the position / orientation of the mobile device 200). In some examples, a determination of a shift in attention (based on a change in head orientation) may be made based on data provided by a position and / or orientation sensor of the wearable device 100 in combination with image data obtained by the image sensor 116 of the wearable device 100. That is, objects, patterns, etc. detected in the image data (such as, for example, in this example, detection of a side-view mirror 525 in the image data) may be used to verify that the user has (temporarily) turned his / her head to check a blind spot, change lanes, etc.

[0069] In some examples, it can be determined based on data provided by the position and / or orientation sensor of the wearable device 100 and / or the position and / or orientation sensor of the mobile device 200 and / or the image data provided by the image sensor 116 of the wearable device 100 that the user has turned his / her head back again to return to the focus through the windshield 555 (i.e., has moved from Figure 5E The position shown is transferred back to Figure 5C or Figure 5D Once such a return of attention to the front of the vehicle through the windshield 555 is detected, the system can again output the following: Figure 5C and / or Figure 5D The information shown.

[0070] Figure 5D and Figure 5E It is shown that the system can adapt or modify the display of information based on the context and surrounding environment in which the wearable device 100 is used. Figure 5D and Figure 5E The system is shown to delay or temporarily suspend the output of information based on the context and surroundings in which the wearable device 100 is used. In some examples, the system may allow the output of some types of information and delay the output of other types of information. The determination of whether to allow the output of information or delay the output of information may be associated with the context and surroundings in which the wearable device 100 is used and the relevance or applicability of the information to be output relative to the context and surroundings. FIG. 5A to FIG. 5E In the example scenario shown, the system can determine that information in the form of route guidance is relevant or applicable to the current use of the wearable device 100, and can output the information when it is determined that the output of the information (e.g., in the form of virtual objects 570, 571) does not cause undue distraction and / or create a safety hazard.

[0071] exist FIG. 5A to FIG. 5E In the example scenario shown, the system may determine that another type of information (such as, for example, a notification of an incoming message) may not be relevant or applicable to the current use of the mobile device 100 and / or may cause unnecessary distraction when received. In such a scenario, the system may delay the output of the information (i.e., the display of the notification of the incoming message) until it is determined that the output of the information will not cause excessive distraction and / or endanger the safety of the situation. For example, the system may determine that the vehicle has stopped based on, for example, positioning data provided by the wearable device 100 and / or the mobile device 200, position and / or orientation data provided by the wearable device and / or the mobile device 200, and may output other types of information (notifications, updates, information requested by the user, etc.) until it is determined that the vehicle is in motion again. The system may determine that the user of the wearable device 100 is in the passenger seat (rather than the driver's seat) based on, for example, image data provided by the wearable device 100, and may output information to the user based on different parameters than when in the driver's seat.

[0072] FIG. 4A to FIG. 4D An application of the principles described herein to the use of an example head-mounted wearable device 100 while walking is shown. FIG. 5A to FIG. 5E The application of the principles described herein to the use of an example head mounted display device 100 while driving is shown. The principles described herein can be applied to the use of the example head mounted wearable device 100 in other environments where reduced distraction can improve the user experience and enhance situational safety. Similarly, the principles described herein can be applied to the use of other types of wearable devices in the described example environments and other environments where reduced user distraction can improve the user experience and enhance situational safety.

[0073] FIG. 4A to FIG. 4D as well as FIG. 5A to FIG. 5E Examples are presented of modifying and / or (temporarily) pausing the output of visual content based on analysis of detected safety factors and concerns in a particular context in which the wearable device 100 is used. FIG. 4A to FIG. 5E In the examples shown, the output of content is modified (e.g., simplified) or temporarily paused to reduce distraction in scenarios where situational safety considerations require such modification / pause of the output of visual content. In some examples, visual content can be replaced with audible content to reduce distraction that may result from outputting visual content in a particular scenario. For example, in a scenario where the output of visual content, even in a modified / simplified manner, may obscure safety-critical features in the user's field of view, audible content may be output in place of visual content.

[0074] Figure 66 is a flow chart of an example method 600 for managing content output in a wearable device according to implementations described herein. As described above, content can be output to a user by a wearable device such as the head-mounted wearable device 100 described above based on a context associated with the use of the wearable device to reduce distraction and maintain situational awareness and situational safety during use of the wearable device. For purposes of discussion and illustration, the example method 600 shows operations performed sequentially. The operations may be performed in a different order than shown, or in a parallel or overlapping manner.

[0075] While the wearable device is operating, data associated with the wearable device may be collected (block 610). The data may include, for example, positioning data associated with the wearable device, location and / or orientation data associated with the wearable device, image data obtained by an image sensor of the wearable device, images capturing the environment in which the wearable device is operating, and other such data. If it is determined that a mobile device such as the handheld mobile computing device 200 described above is paired or operably coupled with the wearable device (block 615), data associated with the mobile device may also be collected (block 620). The data may similarly include positioning data associated with the mobile device, location and / or orientation data associated with the mobile device, image data obtained by an image sensor of the mobile device, and other such data.

[0076] A context associated with use of the wearable device may be determined based on data associated with the wearable device alone or in combination with data associated with the mobile device (block 625). The context may provide, for example, a characterization (e.g., identification and / or classification) of the environment in which the wearable device is operating and / or a current operating mode of the wearable device, etc. The context is associated with safety factors to be considered in the operation of the wearable device in the current environment. In response to determining that content is available for output by the wearable device (block 630) and that the content is relevant to or applicable to the determined context (block 635), the wearable device may output the content (block 645). For example, this corresponds to a first content output state. The content may be output in an allowable or designated content display area of ​​the wearable device, which will allow the content to be output with minimal distraction to the user, minimal obstruction of the user's view of the surrounding environment, etc.

[0077] In response to determining that the content is not relevant to the determined context, the output of the content can be delayed (box 640) to minimize unnecessary distractions. For example, this corresponds to a second content output state. The output of the content can be delayed for a predetermined period of time, a period of time determined by the wearable device and / or the mobile device, and / or until a change to the first content output state. In response to the detected context change (box 650), the form and / or manner of outputting the content can be modified (box 655). The detected content change can indicate a change in user focus, thereby driving the need for a change in how the content is output. Modifications to the form and / or manner of outputting the content can include, for example, visual changes in how the content is output, the level of detail included in the output of the content, changes in the output location of the content, pauses in the output of the content, and the like. The process can be repeated until the use session of the wearable device is terminated (box 660).

[0078] Figure 7 An example of a computer device 700 and a mobile computer device 750 that can be used with the techniques described herein is shown. The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 connected to the memory 704 and a high-speed expansion port 710, and a low-speed interface 712 connected to a low-speed bus 714 and the storage device 706. Each of the components 702, 704, 706, 708, 710, and 712 are interconnected using various buses and can be mounted on a common motherboard or otherwise installed where appropriate. The processor 702 can process instructions for execution within the computing device 700, including instructions stored in the memory 704 or on the storage device 706, to display graphical information of a GUI on an external input / output device (such as a display 716 coupled to the high-speed interface 708). In other implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple types of memories where appropriate. Furthermore, multiple computing devices 700 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0079] The memory 704 stores information within the computing device 700. In one implementation, the memory 704 is one or more volatile memory units. In another implementation, the memory 704 is one or more non-volatile memory units. The memory 704 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0080] The storage device 706 can provide mass storage for the computing device 700. In one implementation, the storage device 706 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory or another similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 704, the storage device 706, or a memory on the processor 702.

[0081] The high-speed controller 708 manages bandwidth-intensive operations of the computing device 700, while the low-speed controller 712 manages less bandwidth-intensive operations. This allocation of functions is merely an example. In one implementation, the high-speed controller 708 is coupled to the memory 704, the display 716 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 710 that can accept various expansion cards (not shown). In an implementation, the low-speed controller 712 is coupled to the storage device 706 and the low-speed expansion port 714. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device (such as a switch or router), for example, through a network adapter.

[0082] The computing device 700 can be implemented in a variety of different forms, as shown. For example, it can be implemented as a standard server 720, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 724. In addition, it can also be implemented in a personal computer (such as a laptop computer 722). Alternatively, components from the computing device 700 can be combined with other components in a mobile device (not shown) (such as device 750). Each of such devices can contain one or more of the computing devices 700, 750, and the entire system can be composed of multiple computing devices 700, 750 that communicate with each other.

[0083] Computing device 750 includes processor 752, memory 764, input / output devices (such as display 754), communication interface 766 and transceiver 768, among other components. Device 750 may also be provided with storage devices such as micro drives or other devices to provide additional storage. Each of components 750, 752, 764, 754, 766 and 768 are interconnected using various buses, and several of these components may be mounted on a common motherboard or otherwise where appropriate.

[0084] Processor 752 can execute instructions within computing device 750, including instructions stored in memory 764. The processor can be implemented as a chipset including a separate chip with multiple analog and digital processors. For example, the processor can provide coordination of other components of device 750, such as control of a user interface, applications executed by device 750, and wireless communications performed by device 750.

[0085] The processor 752 can communicate with the user through the control interface 758 and the display interface 756 coupled to the display 754. The display 754 can be, for example, a TFT LCD (thin film transistor liquid crystal display) and an LED (light emitting diode) or OLED (organic light emitting diode) display or other appropriate display technology. The display interface 756 may include an appropriate circuit system for driving the display 754 to present graphical information and other information to the user. The control interface 758 can receive commands from the user and convert them for submission to the processor 752. In addition, an external interface 762 in communication with the processor 752 can be provided to enable near-field communication of the device 750 with other devices. In some implementations, the external interface 762 can provide, for example, wired communication, or in other implementations, wireless communication, and multiple interfaces can also be used.

[0086] The memory 764 stores information within the computing device 750. The memory 764 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 774 may also be provided and connected to the device 750 via an expansion interface 772, which may include, for example, a SIMM (single in-line memory module) card interface. Such an expansion memory 774 may provide additional storage space for the device 750, or may also store applications or other information of the device 750. Specifically, the expansion memory 774 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Therefore, for example, the expansion memory 774 may be provided as a security module of the device 750, and may be programmed with instructions that permit the secure use of the device 750. In addition, security applications and additional information may also be provided via a SIMM card, such as placing identification information on a SIMM card in an unbreakable manner.

[0087] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or machine-readable medium, such as memory 764, expansion memory 774, or memory on processor 752, which may be received, for example, via transceiver 768 or external interface 762.

[0088] The device 750 can communicate wirelessly via a communication interface 766, which may include a digital signal processing circuit system when necessary. The communication interface 766 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. Such communication can occur, for example, via a radio frequency transceiver 768. In addition, short-range communication can occur, such as using Bluetooth, Wi-Fi or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 770 can provide additional navigation and positioning related wireless data to the device 750, which can be used by applications running on the device 750 where appropriate.

[0089] Device 750 may also communicate audibly using audio codec 760, which may receive verbal information from a user and convert it into usable digital information. Audio codec 760 may also generate audible sounds for the user, such as through a speaker (e.g., in an earpiece of device 750). Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 750.

[0090] The computing device 750 can be implemented in many different forms, as shown in the figure. For example, it can be implemented as a cellular phone 780. It can also be implemented as part of a smart phone 782, a personal digital assistant or other similar mobile device.

[0091] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0092] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium," "computer-readable medium," refer to any computer program product, apparatus, and / or device (e.g., a disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (LED (light emitting diode), or OLED (organic LED), or LCD (liquid crystal display) monitor / screen) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, voice, or tactile input.

[0094] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0095] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0096] In some implementations, the computing device depicted in the figure may include a sensor that interfaces with the AR head-mounted device / HMD device 790 to generate an augmented environment for viewing inserted content in the physical space. For example, one or more sensors included on the computing device 750 depicted in the figure or other computing devices may provide input to the AR head-mounted device 790, or generally speaking, to the AR space. Sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 750 may use sensors to determine the absolute position and / or detected rotation of the computing device in the AR space, and then the absolute position and / or the detected rotation may be used as input to the AR space. For example, the computing device 750 may be incorporated into the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, etc. The positioning of the computing device / virtual object by the user when incorporated into the AR space may allow the user to position the computing device so as to view the virtual object in some way in the AR space. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device as if it were an actual laser pointer. The user can move the computing device left to right, up and down, in circles, etc., and use the device in a manner similar to using a laser pointer. In some implementations, the user can aim at a target using the virtual laser pointer.

[0097] In some implementations, one or more input devices included on or connected to the computing device 750 can be used as input to the AR space. The input device can include, but is not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, headphones or earbuds with input functionality, a game controller, or other connectable input device. When the computing device 750 is incorporated into the AR space, a user interacting with the input device included on the computing device can cause specific actions to occur in the AR space.

[0098] In some implementations, the touch screen of the computing device 750 can be rendered as a touch pad in the AR space. The user can interact with the touch screen of the computing device 750. For example, in the AR head mounted device 790, the interaction is rendered as movement on the rendered touch pad in the AR space. The rendered movement can control the virtual object in the AR space.

[0099] In some implementations, one or more output devices included on the computing device 750 can provide output and / or feedback to a user of the AR head mounted device 790 in the AR space. The output and feedback can be visual, tactile, or auditory. The output and / or feedback can include, but are not limited to, vibration, turning on and off or blinking and / or flashing one or more lights or flashes, sounding an alarm, playing a chime, playing a song, and playing an audio file. The output device can include, but is not limited to, a vibration motor, a vibration coil, a piezoelectric device, an electrostatic device, a light emitting diode (LED), a flash, and a speaker.

[0100] In some implementations, computing device 750 may appear as another object in a computer-generated 3D environment. Interaction by a user with computing device 750 (e.g., rotating, shaking, touching a touch screen, sliding a finger across a touch screen) may be interpreted as interaction with an object in AR space. In the example of a laser pointer in AR space, computing device 750 appears as a virtual laser pointer in a computer-generated 3D environment. As the user manipulates computing device 750, the user in AR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 750 in the AR environment on computing device 750 or AR head-mounted device 790. User interaction with the computing device can be converted into interaction with a user interface generated for a controllable device in the AR environment.

[0101] In some implementations, computing device 750 may include a touch screen. For example, a user may interact with the touch screen to interact with a user interface of a controllable device. For example, the touch screen may include user interface elements, such as sliders, that may control properties of the controllable device.

[0102] Computing device 700 is intended to represent various forms of digital computers and devices, including but not limited to laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 750 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be examples only, and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0103] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the present description.

[0104] In addition, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve the desired results. In addition, other steps can be provided, or steps can be deleted from the described processes, and other components can be added to or removed from the described systems. Therefore, other embodiments are within the scope of the appended claims.

[0105] In addition to the above, controls may be provided to allow the user to make a choice about whether and when the systems, programs, or features described herein may be able to collect user information (e.g., information about the user's social network, social behavior, or activities, occupation, user preferences, or the user's current location) and whether to send content or communications from a server to the user. In addition, before certain data is stored or used, it may be processed in one or more ways to remove personally identifiable information. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location for which location information is obtained may be generalized (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0106] Although certain features of the described implementations have been shown as described herein, those skilled in the art will now expect many modifications, substitutions, variations and equivalents. Therefore, it should be understood that the appended claims are intended to cover all these modifications and variations that fall within the scope of the implementations. It should be understood that they are presented only in an exemplary and non-restrictive manner, and various changes in form and detail may be made. Except for mutually exclusive combinations, any part of the devices and / or methods described herein may be combined in any combination. The implementations described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations described.

Claims

1. A computer-implemented method comprising: Determining a context associated with operation of the head mounted wearable device; determining a content output state of the head-mounted wearable device based on the context and one or more situational safety factors associated with the context; as well as Outputting content by a display device of the head-mounted wearable device based on the content output state of the head-mounted wearable device includes: In a first content output state, outputting the content in a display area of ​​the head-mounted wearable device; as well as In the second content output state, the output of the content is delayed.

2. The computer-implemented method of claim 1 , wherein determining the context comprises: Obtaining data associated with the head-mounted wearable device, the data comprising at least one of the following: Positioning data provided by the head-mounted wearable device; Image data provided by an image sensor of the head-mounted wearable device; or Position and orientation data provided by a position and orientation sensor of the head-mounted wearable device.

3. A computer-implemented method as described in claim 2, wherein obtaining the data associated with the head-mounted wearable device includes obtaining at least one of positioning data, image data, position data, or orientation data from a mobile computing device paired with the head-mounted wearable device.

4. The computer-implemented method of any preceding claim, further comprising: Determining the suitability of the content for the context; outputting the content in response to determining that the content is applicable to the context; as well as Outputting of the content is delayed in response to determining that the content is not applicable to the context.

5. The computer-implemented method of any one of the preceding claims, wherein outputting the content comprises outputting the content in a designated content display area within the display area of ​​the head-mounted wearable device, comprising: Identify a portion of a display area of ​​the head-mounted wearable device as the designated content display area; Display the content only in the designated content display area; designating a remaining portion of the display area of ​​the head mounted wearable device as a non-display area; as well as Display of content is restricted in the non-display area.

6. The computer-implemented method of claim 5, wherein designating the remaining portion of the display area of ​​the head mounted wearable device as a non-display area comprises: designating a portion of the display area of ​​the head-mounted wearable device corresponding to the one or more situational safety factors as a non-display area; as well as Display of content is restricted in the non-display area, thereby maintaining a user's field of view through the non-display area.

7. The computer-implemented method of any preceding claim, further comprising: detecting a change in the context in which the head mounted wearable device is operating; as well as The content output state of the head-mounted wearable device is updated based on the changed context.

8. The computer-implemented method of claim 7, further comprising: The manner in which the content is output is modified based on the changed context and the updated content output state.

9. The computer-implemented method of claim 8, wherein modifying the manner in which the content is output comprises at least one of: simplifying the appearance of the content in response to a detected change in the one or more contextual security factors; changing a display location of the content in response to a detected change in the one or more contextual security factors; pausing output of the content in response to a detected change in the one or more contextual security factors; or Output of the content is resumed in response to the detected change in the one or more contextual security factors.

10. A computer-implemented method as described in any of the preceding claims, wherein the first content output state is based on the context in which the head-mounted wearable device operates and the one or more situational safety factors associated with the operation of the head-mounted wearable device, and the content output in the designated content display area maintains a situational safe state.

11. A computer-implemented method as described in any of the preceding claims, wherein the second content output state is based on the context in which the head-mounted display operates and the one or more situational safety factors associated with the operation of the head-mounted wearable device, and the delay in the content output maintains a situational safety state.

12. A non-transitory computer-readable medium storing executable instructions that, when executed by at least one processor of a computing device, are configured to cause the at least one processor to: Determining a context associated with operation of the head mounted wearable device; determining a content output state of the head-mounted wearable device based on the context and one or more situational safety factors associated with the context; and Outputting content by a display device of the head-mounted wearable device based on the content output state of the head-mounted wearable device includes: In a first content output state, outputting the content in a display area of ​​the head-mounted wearable device; as well as In the second content output state, the output of the content is delayed.

13. The non-transitory computer readable medium of claim 12, wherein the instructions cause the at least one processor to: Obtaining data associated with the head-mounted wearable device, the data comprising at least one of the following: Positioning data provided by at least one of the head-mounted wearable device or a mobile computing device paired with the head-mounted wearable device; Image data provided by an image sensor of the head-mounted wearable device; or Position and orientation data provided by a position and orientation sensor of at least one of the head-mounted wearable device or the mobile computing device.

14. The non-transitory computer readable medium of claim 12 or 13, wherein the instructions cause the at least one processor to: Determining the suitability of the content for the context; outputting the content in response to determining that the content is applicable to the context; and Outputting of the content is delayed in response to determining that the content is not applicable to the context.

15. The non-transitory computer readable medium of any one of claims 12 to 14, wherein the instructions cause the at least one processor to: Identify a portion of the display area of ​​the head-mounted wearable device as a designated content display area; Display the content only in the designated content display area; designating a remaining portion of the display area of ​​the head mounted wearable device as a non-display area; and Display of content is restricted in the non-display area.

16. The non-transitory computer readable medium of claim 15, wherein the instructions cause the at least one processor to: designating a portion of the display area of ​​the head-mounted wearable device corresponding to the one or more situational safety factors as a non-display area; and Display of content is restricted in the non-display area, thereby maintaining a user's field of view through the non-display area.

17. The non-transitory computer readable medium of any one of claims 12 to 16, further comprising: detecting a change in the context in which the head mounted wearable device is operating; updating the content output state of the head-mounted wearable device based on the changed context; as well as The manner in which the content is output is modified based on the changed context and the updated content output state.

18. The non-transitory computer-readable medium of claim 17, wherein the instructions cause the at least one processor to modify the manner in which the content is output, comprising: simplifying the appearance of the content in response to a detected change in the one or more contextual security factors; changing a display location of the content in response to a detected change in the one or more contextual security factors; pausing output of the content in response to a detected change in the one or more contextual security factors; or Output of the content is resumed in response to the detected change in the one or more contextual security factors.

19. A non-transitory computer-readable medium as described in any one of claims 12 to 18, wherein the first content output state is based on the context in which the head-mounted wearable device operates and the one or more situational safety factors associated with the operation of the head-mounted wearable device, and the content output in the display area maintains a situational safe state.

20. A non-transitory computer-readable medium as described in any one of claims 12 to 19, wherein the second content output state is based on the context in which the head-mounted display operates and the one or more situational safety factors associated with the operation of the head-mounted wearable device, and the delay in the content output maintains a situational safety state.

Images