Augmented reality system and method

Abstract

The present disclosure provides "augmented reality systems and methods". A computer includes a processor and a memory, and the memory stores instructions executable by the processor to determine, based on image data of a user depicting a display screen, that the user is wearing augmented reality glasses; and in response to determining that the user is wearing the augmented reality glasses and in response to identifying a portion of content for display by the display screen as including restricted data, displaying the portion with the augmented reality glasses and omitting the portion to be displayed on the display screen. The display screen is separated from the augmented reality glasses.

Description

Technical Field

[0001] This disclosure relates to augmented reality systems. Background Technology

[0002] Modern vehicles typically include an audiovisual system, which may include displays and speakers. The displays can be of various types suitable for displaying readable content for vehicle occupants, such as light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), liquid crystal displays (LCDs), plasma displays, and digital light processing technology (DLPT). Speakers can be installed in different locations within the passenger compartment to project sound towards the vehicle's occupants. Summary of the Invention

[0003] This disclosure provides techniques for an augmented reality system to selectively display portions of content via different devices to keep certain data restricted to a specific user. The devices are a display screen and augmented reality (AR) glasses. The display screen is separate from the AR glasses; for example, the display screen may be a user's personal device or a portion of a vehicle occupied by the user. A computer is programmed to determine, based on image data depicting the user, that the user is wearing the AR glasses. Certain content is set to be displayed on the display screen. The computer is also programmed to, in response to determining that the user is wearing the AR glasses and in response to recognizing a portion of the content as including restricted data, display the restricted portion using the AR glasses and prevent the restricted portion from being displayed on the display screen. In other words, the computer can reroute a portion of the content identified as including restricted data to the AR glasses, and the computer can maintain the remaining portion of the content on the display screen. Using image data depicting the user can help prevent restricted data from being unintentionally displayed, such as if the user forgets to manually select the data to be displayed on the AR glasses, thus maintaining the restricted state of the data. These techniques may be particularly useful in the context of vehicles (such as cars, trains, buses, etc.). In these contexts, occupants may be huddled together, potentially allowing another occupant to see the user's display screen. For example, the computer could be the vehicle's onboard computer, and an onboard camera could provide image data depicting the user.

[0004] A computer includes a processor and a memory, the memory storing instructions executable by the processor to determine, based on image data of a user depicting a display screen, that the user is wearing augmented reality glasses; and, in response to determining that the user is wearing the augmented reality glasses and in response to identifying a portion of content to be displayed on the display screen as including restricted data, to display the portion with the augmented reality glasses and omit the portion from the display screen. The display screen is separate from the augmented reality glasses.

[0005] In the example, the instructions may also include instructions to: identify the user based on the image data, and in response to determining that the user is wearing the augmented reality glasses and identifying the user as a pre-stored user, display the portion with the augmented reality glasses, and prevent the portion from being displayed on the display screen.

[0006] In this example, the instructions may also include instructions for receiving image data from the camera. In another example, the vehicle occupied by the user may include a camera.

[0007] In another example, the personal device being used by the user may include a camera and a display screen.

[0008] In another example, the personal device being used by the user may include a display screen. In yet another example, the instructions may also include instructions for pairing with the personal device and the augmented reality glasses.

[0009] In the example, the instructions may also include instructions to display the second portion on the display screen in response to recognizing that the second portion of the content does not include restricted data.

[0010] In the example, the instructions may also include instructions to: in response to determining that the user is not wearing the augmented reality glasses, output a message prompting the user to put on the augmented reality glasses.

[0011] In the example, the instructions may also include instructions to lock the display screen in response to determining that the user has removed the augmented reality glasses.

[0012] In the example, the instructions may also include instructions to: in response to input from the user, display a second portion of the content using the augmented reality glasses and prevent the second portion from being displayed on the screen.

[0013] In the example, the instructions may also include instructions to identify the portion of the content as including restricted data based on the fact that the portion of the content includes the user's personally identifiable information.

[0014] In the example, the instructions also include instructions for identifying the portion of the content as containing restricted data based on the data encryption tag associated with that portion.

[0015] In the example, the instructions may also include instructions to identify the portion of the content as including restricted data based on the fact that the portion of the content includes age-restricted material.

[0016] A method includes determining, based on image data of a user depicting a display screen, that the user is wearing augmented reality glasses; and, in response to determining that the user is wearing the augmented reality glasses and in response to identifying a portion of content to be displayed on the display screen as including restricted data, displaying the portion with the augmented reality glasses and omitting the portion from the display screen. The display screen is separate from the augmented reality glasses.

[0017] In the example, the method may further include: identifying the user based on the image data, and in response to determining that the user is wearing the augmented reality glasses and identifying the user as a pre-stored user, displaying the portion with the augmented reality glasses, and preventing the portion from being displayed on the display screen.

[0018] In one example, the method may further include receiving the image data from a camera. In another example, a vehicle occupied by a user may include a camera.

[0019] In another example, the personal device being used by the user may include a camera and a display screen.

[0020] In the example, the method may further include: displaying the second portion on the display screen in response to identifying the second portion of the content as not including restricted data. Attached Figure Description

[0021] Figure 1 This is a side view schematic diagram of a portion of an example vehicle that includes an example augmented reality system.

[0022] Figure 2 This is an illustration of example content displayed on the display screen of an augmented reality system, overlaid on the augmented reality glasses.

[0023] Figure 3 This is a flowchart of an example process for controlling an augmented reality system. Detailed Implementation

[0024] Referring to the accompanying drawings, wherein like reference numerals indicate like parts in several views, computer 105 includes a processor and memory, and the memory stores instructions executable by the processor to determine, based on image data depicting a user on display screen 110, that the user is wearing augmented reality (AR) glasses 115, and, in response to determining that the user is wearing AR glasses 115 and in response to identifying a portion of the content 200 to be displayed on display screen 110 as including restricted data, to display that portion with AR glasses 115 and prevent that portion from being displayed on display screen 110. Display screen 110 is separate from AR glasses 115.

[0025] refer to Figure 1 The AR system 101 may include a computer 105, a display screen 110, and AR glasses 115.

[0026] The AR system 101 described herein can be implemented in vehicle 100. Vehicle 100 can be any passenger or commercial vehicle, such as a sedan, truck, SUV, crossover, van, minivan, taxi, bus, etc. Alternatively, vehicle 100 can be another mode of transportation, such as a train, subway, ship, airplane, etc.

[0027] Vehicle 100 includes a passenger compartment 120 for accommodating occupants of vehicle 100. For automobiles, passenger compartment 120 may include one or more seats 125 disposed in the front row of passenger compartment 120, one or more seats 125 disposed in the second row behind the front row, and / or one or more seats 125 disposed in the third row at the rear of passenger compartment 120. The position and orientation of the seats 125 and their components can be adjusted by the occupants. A user occupies vehicle 100 by sitting in passenger compartment 120 and / or in one of the seats 125.

[0028] A user may have a personal device 130 in vehicle 100. Personal device 130 is a portable computing device, such as a mobile phone (e.g., a smartphone), a laptop computer, or a tablet computer. Personal device 130 is distinct from vehicle 100 and is not a component of vehicle 100. Personal device 130 is a computing device including a processor and memory. Personal device 130 is used and carried by a user who may be the operator of vehicle 100.

[0029] Display screen 110 is separate from AR glasses 115. Display screen 110 can be a component of personal device 130 or vehicle 100. In other words, personal device 130 can include display screen 110, or vehicle 100 can include display screen 110. For example, display screen 110 can be the screen of a mobile phone, laptop computer, or tablet computer. Alternatively, display screen 110 can be fixedly mounted in the passenger compartment 120 of vehicle 100 (e.g., on the center console or dashboard of vehicle 100). Display screen 110 can be any suitable type for displaying user-readable content 200, such as light-emitting diode (LED), organic light-emitting diode (OLED), liquid crystal display (LCD), plasma, digital light processing technology (DLPT), etc.

[0030] A user may have AR glasses 115 in vehicle 100. AR glasses 115 can be worn on the user's face and extend in front of the user's eyes. AR glasses 115 provide the user with augmented reality, which, from the user's perspective, combines the real world with computer-generated content superimposed on the real world. AR glasses 115 have transparent lenses that allow the user to see their surroundings. The lenses of AR glasses 115 also display content, which the user sees while viewing the world through the lenses. AR glasses 115 can display content on the lenses at a specific size and position, such that the content appears to the user to be located in a specific location in the world. For example, AR glasses 115 can use pose tracking to determine the position and orientation of the user's head relative to the surrounding environment (e.g., based on data from an inertial measurement unit (IMU), etc.).

[0031] Computer 105 is a microprocessor-based computing device, such as a general-purpose computing device (including a processor and memory, electronic controller, etc.), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a combination thereof. Typically, hardware description languages ​​such as VHDL (VHSIC (Very High Speed ​​Integrated Circuit) Hardware Description Language) are used in electronic design to describe digital and mixed-signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided before manufacturing, while the logic components inside an FPGA can be configured based on VHDL programming (e.g., stored in memory electrically connected to the FPGA circuitry). Therefore, computer 105 may include a processor, memory, etc. The memory of computer 105 may include media for storing instructions executable by the processor and for electronically storing data and / or databases, and / or computer 105 may include structures such as those providing programming capabilities. Computer 105 may consist of multiple computers coupled together.

[0032] Computer 105 may be a component of vehicle 100 or personal device 130. In other words, vehicle 100 may include computer 105, or personal device 130 may include computer 105. For example, computer 105 may be an onboard computer of vehicle 100 and fixedly installed inside vehicle 100. Alternatively, computer 105 may be a processor and memory for operating personal device 130.

[0033] AR system 101 may include camera 135. Camera 135 can detect electromagnetic radiation within a certain wavelength range. For example, camera 135 can detect visible light, infrared radiation, ultraviolet light, or a range of wavelengths including visible light, infrared light, and / or ultraviolet light. For example, camera 135 may be a charge-coupled device (CCD), complementary metal-oxide-semiconductor (CMOS), or any other suitable type.

[0034] Camera 135 may be a component of vehicle 100 or personal device 130. In other words, vehicle 100 may include camera 135, or personal device 130 may include camera 135. For example, camera 135 may be fixedly mounted within the passenger compartment 120 of vehicle 100 (e.g., at the front of passenger compartment 120 and rearwardly oriented, such as...). Figure 1 (As shown). Alternatively, camera 135 can be mounted to personal device 130. In this case, camera 135 can be oriented to face the same direction as the display screen 110 of personal device 130 (i.e., towards the user's face when the user is reading the display screen 110 of personal device 130).

[0035] Computer 105 is programmed to receive image data from camera 135. The image data is a series of image frames within the field of view of camera 135. Each image frame is a two-dimensional matrix of pixels. The brightness or color of each pixel is represented as one or more numerical values, such as a scalar, unitless value of luminance intensity between 0 (black) and 1 (white), or a value for each of red, green, and blue (e.g., each using an 8-bit scale (0 to 255) or a 12-bit or 16-bit scale). Pixels can be a mixture of representations (e.g., a repeating pattern of three pixels and a fourth pixel with scalar values ​​of intensity having three numerical color values, or some other pattern). The position within an image frame (i.e., the position within the field of view of camera 135 at the time the image frame is recorded) can be specified by pixel size or coordinates (e.g., a pair of ordered pixel distances), such as a number of pixels from the top edge of the image frame and a number of pixels from the left edge of the image frame.

[0036] Vehicle 100 may include a communication network 140. As a component of vehicle 100, computer 105 can send and receive data via communication network 140. Communication network 140 may be a controller local area network (CAN) bus, Ethernet, Wi-Fi, local area internet (LIN), on-board diagnostic connector (OBD-II), and / or any other wired or wireless communication network. Computer 105 may be communicatively coupled to camera 135 (if part of vehicle 100), display screen 110 (if part of vehicle 100), transceiver 145, user interface 150, and other components via communication network 140.

[0037] Vehicle 100 may include transceiver 145. Transceiver 145 may be adapted to communicate via any suitable wireless communication protocol (such as cellular, etc.). The transceiver 145 can wirelessly transmit signals using technologies such as Broadband Low Energy (BLE), Ultra Wideband (UWB), Wi-Fi, IEEE 802.11a / b / g / p, Cellular V2X (CV2X), Dedicated Short Range Communication (DSRC), and other RF (radio frequency) communications. The transceiver 145 can be used to communicate with a remote server (i.e., a server that is different from and spaced apart from vehicle 100). For example, the remote server can be associated with another vehicle (e.g., V2V communication), with infrastructure components (e.g., V2I communication), with a first responder, with a user's personal device 130, with AR glasses 115, and so on. The transceiver 145 can be a single device or may include separate transmitters and receivers.

[0038] Vehicle 100 may include a user interface 150. The user interface 150 presents information to and receives information from the occupants of vehicle 100. The user interface 150 may be located on a dashboard in the passenger compartment 120 of vehicle 100, and / or anywhere easily visible to the occupants. The user interface 150 may include dials, digital readout devices, screens, speakers, etc., for providing information to the occupants, such as known human-machine interface (HMI) elements. For example, the user interface 150 may include a display screen 110 of vehicle 100. The user interface 150 may include buttons, knobs, keypads, microphones, etc., for receiving information from the occupants.

[0039] Computer 105 can be programmed to receive user input from a user via user interface 150 and / or personal device 130. For example, the user can use buttons or keys, a touchscreen, a trackpad, etc., on personal device 130 or user interface 150. As another example, the user can provide audible voice commands to the microphone of personal device 130 or user interface 150, and computer 105 can interpret the voice commands using known algorithms. As another example, the user can provide gesture commands. Computer 105 can receive image data from camera 135 depicting the user performing a gesture command, and computer 105 can interpret the gesture commands using known algorithms. The image data can come from the same camera 135 that generates image data used to determine whether the user is wearing AR glasses 115 (described below).

[0040] Computer 105 can be programmed to pair with personal device 130 and / or AR glasses 115. For the purposes of this disclosure, "pairing" is defined as establishing a connection that permits communication between two devices. Computer 105 may use transceiver 145 to establish the connection. The connection may include negotiation between transceiver 145 and personal device 130 or AR glasses 115. Each negotiation identifies transceiver 145 to personal device 130 or AR glasses 115, and then identifies personal device 130 or AR glasses 115 to transceiver 145, thereby permitting the transmission of signals between transceiver 145 and personal device 130 or AR glasses 115. The connection may use any suitable protocol (e.g., The connection can be encrypted or unencrypted.

[0041] Computer 105 can be programmed to recognize users (i.e., determine if a user is a pre-stored user, in other words, a known specific user with an identity stored in the memory of computer 105). Computer 105 can recognize users based on image data. Computer 105 can perform facial recognition on image data to determine whether a face depicted in the image data is an identified face (i.e., the face of a known individual such as the operator of vehicle 100 stored in memory). Computer 105 can use any suitable facial recognition technique, such as template matching; statistical techniques, such as principal component analysis (PCA), discrete cosine transform, linear discriminant analysis, local preserving projection, Gabor wavelet, independent component analysis, or kernel PCA; neural networks, such as neural networks with Gabor filters, neural networks with Markov models, or fuzzy neural networks; etc.

[0042] Alternatively, computer 105 can identify the user based on another input. For example, sensors in AR glasses 115 or personal device 130 (e.g., an iris scanner built into AR glasses 115) can provide biometric authentication for the user. As another example, the user can use a key fob, etc., to start vehicle 100, and the key fob may have an RFID tag, etc., that uniquely identifies the user from other potential operators who frequently use vehicle 100. The RFID signal can be associated with the user in memory. As another example, computer 105 can identify the user based on the pairing of personal device 130 or AR glasses 115 with computer 105. Personal device 130 or AR glasses 115 can be associated with the user in memory. As another example, the operator can input identification information (such as a username and password or master key) into user interface 150.

[0043] Computer 105 can be programmed to prompt the user to create a profile in response to recognizing the user as not a pre-stored user (i.e., in response to the user not being identified). The user can specify a restricted data policy (described below) as part of the profile. The profile can be stored in memory. Once the profile is created, computer 105 can recognize the user as a pre-stored user the next time the user attempts to use vehicle 100.

[0044] Computer 105 is programmed to determine whether a user is wearing AR glasses 115 based on image data depicting a user on display screen 110. Computer 105 can detect the user's face and then classify the face as either wearing AR glasses 115 or not. Computer 105 may rely on a facial recognition algorithm already implemented for face detection or may use facial detection techniques such as: knowledge-based techniques, such as multi-resolution rule-based methods; feature-invariant techniques, such as edge grouping, spatial gray-level correlation matrix, or Gaussian mixture; template matching techniques, such as shape templates or active shape models; or appearance-based techniques, such as intrinsic face decomposition and clustering, Gaussian distributions and multilayer perceptrons, neural networks, support vector machines with multinomial kernels, Naive Bayes classifiers with joint statistics of local appearance and location, higher-order statistics with hidden Markov models, or Kullback-related information.

[0045] Upon detecting a face, computer 105 can classify the face as either wearing AR glasses 115 or not wearing AR glasses 115. Computer 105 can employ conventional image recognition techniques, such as convolutional neural networks (CNNs), which are programmed to accept an image depicting a face as input and output a classification of the face. The CNN comprises a series of layers, each using the previous layer as input. Each layer contains multiple neurons that receive data generated by a subset of the neurons in the previous layer as input and generate an output that is passed to the neurons in the next layer. The types of layers include: convolutional layers that compute the dot product of weights and small regions of input data; pooling layers that perform downsampling operations along the spatial dimension; and fully connected layers that generate based on the outputs of all neurons in the previous layer. The final layer of the CNN generates a score for each potential classification (in this case, wearing AR glasses 115 or not wearing AR glasses 115), and the final output is the classification with the highest score.

[0046] Computer 105 can be programmed to output a message prompting the user to put on AR glasses 115 in response to determining that the user is not wearing AR glasses 115. For example, computer 105 can output the message via user interface 150 (e.g., via display screen 110). This message can ask the user if they want to wear AR glasses 115. The output of this message can also be conditional upon recognizing the user as a pre-stored user. In other words, computer 105 can be programmed to output this message in response to determining that the user is not wearing AR glasses 115 and recognizing the user as a pre-stored user (e.g., a pre-stored user who owns AR glasses 115). A user profile stored in memory can indicate that the user owns AR glasses 115.

[0047] Computer 105 can be programmed to, in response to determining that a user is wearing AR glasses 115, display a portion of content 200 including restricted data using AR glasses 115, and prevent said portion from being displayed on display screen 110, as will be explained below. Figure 2 As described above. For example, computer 105 can apply a restricted data policy. The application of a restricted data policy can also be conditional upon identifying the user as a pre-stored user. In other words, computer 105 can be programmed to apply a restricted data policy in response to determining that a user is wearing AR glasses 115 and identifying the user as a pre-stored user. The restricted data policy can be specified in a user profile stored in memory.

[0048] refer to Figure 2 Content 200 can be configured to be displayed on display screen 110. For example, a user can request content 200 by navigating to a website, selecting media to play, opening an app, etc., on display screen 110 of personal device 130 or computer 105. Instead of displaying content 200 entirely on display screen 110, computer 105 can partition the display of content 200 between display screen 110 and AR glasses 115 in response to conditions described below. Computer 105 can identify a first portion 205 of content 200 for display on AR glasses 115 and a second portion 210 of content 200 (e.g., the remainder of content 200) for display on display screen 110, as described below. Figure 2In the example shown, the first part 205 is the username and password entered on the website, and the second part 210 is the rest of the website and other content on display screen 110. Computer 105 displays the first part 205 using AR glasses 115 and prevents the first part 205 from being displayed on display screen 110, while simultaneously displaying the second part 210 on display screen 110. For example, the area on display screen 110 that would be occupied by the first part 205 could be instead blank (e.g., white or a preset background color). AR glasses 115 can display the first part 205 at a position, orientation, and size on lenses corresponding to the position it would occupy if displayed on display screen 110. Figure 2 In the example, the fields for entering the username and password on the website are blank on display 110, and the username and password are displayed by AR glasses 115 so that they are positioned in the fields as seen by the user wearing AR glasses 115.

[0049] Computer 105 can be programmed to identify a portion of content 200 as including restricted data. Computer 105 can identify the portion based on metadata associated with content 200, which tagged the portion with specific tags pre-stored in memory.

[0050] For example, computer 105 may identify a portion of content 200 as containing restricted data based on the fact that that portion includes the user's personally identifiable information. For the purposes of this disclosure, "personally identifiable information" is defined as an information representation that allows a reasonable inference into the identity of the individual to whom the information is applied. Examples of personally identifiable information include usernames; passwords; identification numbers such as Social Security numbers, driver's license or passport numbers or account numbers; addresses; telephone numbers; email addresses; biometric data; and so on.

[0051] In another example, computer 105 may identify a portion of content 200 as containing restricted data based on a data encryption tag associated with that portion. The data encryption tag indicates that certain data sent to computer 105 or personal device 130 is encrypted (e.g., a certificate used for HTTPS (Hypertext Transfer Protocol Secure)). Examples include secure websites, messages sent via encrypted messaging applications, etc.

[0052] In another example, computer 105 may identify a portion of content 200 as containing age-restricted material. Certain media (such as movies or games) may include ratings recommending a minimum age for accessing the media (e.g., issued by the Motion Picture Association of America (MPAA) or the Entertainment Software Rating Board (ESRB), and these ratings may be stored as metadata associated with the media.

[0053] Computer 105 may store restricted data policies. Restricted data policies specify how restricted data is handled, including whether to display restricted data using AR glasses 115 instead of display screen 110. Restricted data policies may include rules for different types of restricted data. Rules may define, for the corresponding type of restricted data, whether to display the portion containing the restricted data on AR glasses 115 or display screen 110 (e.g., displaying personally identifiable information and a movie rated "R" on AR glasses 115 and displaying other content on display screen 110). Restricted data policies may be user-specific and associated with a user profile stored in the memory of computer 105. Users may provide user input to modify the restricted data policy for that user.

[0054] Computer 105 is programmed to apply a restricted data policy to content 200 in response to recognizing a portion of content 200 as including restricted data covered by a restricted data policy. The restricted data policy includes rules specifying restricted data for display on AR glasses 115 instead of display on display screen 110. Applying the restricted data policy includes, for the parts of content 200 covered by the rules, displaying those parts on AR glasses 115 and preventing those parts from being displayed on display screen 110, as described above with respect to section 205. Applying the restricted data policy also includes, for the parts of content 200 not covered by the rules, displaying those parts on display screen 110, as described above with respect to section 210 (unless moved to AR glasses 115 as described below).

[0055] Computer 105 can be programmed to display a portion of content 200 on AR glasses 115 and prevent said portion from being displayed on display screen 110 (as described above) in response to input from a user. The user can provide input in any of the aforementioned ways (e.g., touchscreen, gestures, etc.). The input can (e.g., circling the portion on a touchscreen or using a gesture) indicate which portion of content 200 should be displayed on AR glasses 115. Computer 105 may have determined that said portion lacks restricted content. Therefore, in addition to the portion with restricted data, the user can also select additional portions of content 200 to be displayed on AR glasses 115. Similarly, computer 105 can be programmed to display a portion of content 200 previously displayed on AR glasses 115 on display screen 110 in response to input from a user.

[0056] Computer 105 can be programmed to track the length of time a user is using AR glasses 115. For example, computer 105 can store the time AR glasses 115 have been paired with computer 105 and periodically subtract the stored time from the current time, or computer 105 can start a timer when AR glasses 115 are paired with computer 105. Computer 105 can output a message to the user (e.g., via user interface 150) in response to the duration exceeding a time threshold. The message can indicate that the user has been using AR glasses 115 for an extended period. The time threshold can be selected to be less than the duration of motion sickness that may sometimes occur due to using AR glasses 115.

[0057] Computer 105 can be programmed to, in response to determining that the user has removed AR glasses 115, stop displaying portions of content 200 on AR glasses 115, display those portions on display screen 110, and lock display screen 110. Locking display screen 110 means displaying content covering content 200 on display screen 110 or deactivating display screen 110. Computer 105 can unlock display screen 110 only in response to user input to verify user identity. Locking display screen 110 can prevent portions of content 200 containing restricted data from being displayed on display screen 110, while still giving the user the option to unlock display screen 110 if they wish to continue interacting with content 200.

[0058] Figure 3This is a flowchart illustrating an example process 300 for controlling display screen 110 and AR glasses 115. The memory of computer 105 stores executable instructions for performing the steps of process 300, and / or can be programmed using structures such as those mentioned above. As a general overview of process 300, computer 105 pairs with personal device 130 and AR glasses 115, receives image data from camera 135, and identifies the user of personal device 130. In response to not identifying the user or determining that the user is not wearing AR glasses 115, computer 105 prompts the user to create a profile or put on AR glasses 115 (unless already prompted) and checks again. In response to identifying the user as a pre-stored user and determining that the user is wearing AR glasses 115, computer 105 initiates AR glasses 115 and loads the user's restricted data policy. In response to identifying a portion of content 200 as including restricted data, computer 105 displays said portion with AR glasses 115 and prevents said portion from being displayed on display screen 110, and computer 105 displays the remaining portion on display screen 110. In response to user input, computer 105 displays a portion of content 200 on AR glasses 115 and prevents said portion from being displayed on display screen 110 (or vice versa). In response to AR glasses 115 usage time exceeding a threshold, computer 105 outputs a message to the user. In response to determining that the user has removed AR glasses 115, computer 105 stops displaying portions of content 200 on AR glasses 115, displays those portions on display screen 110, and locks display screen 110. Process 300 continues until the user unpairs or turns off personal device 130.

[0059] Process 300 begins in box 305, where computer 105 is paired with personal device 130 and AR glasses 115, as described above.

[0060] Next, in box 310, computer 105 receives image data from camera 135, as described above.

[0061] Next, in box 315, computer 105 identifies the user (e.g., based on image data), as described above.

[0062] Next, in decision box 320, computer 105 determines whether the user is wearing AR glasses 115 based on the image data depicting the user from box 310, and determines whether the user is a pre-stored user. In response to not recognizing the user or determining that the user is not wearing AR glasses 115, process 300 proceeds to decision box 325. In response to recognizing the user as a pre-stored user and determining that the user is wearing AR glasses 115, process 300 proceeds to box 335.

[0063] In decision box 325, computer 105 determines whether it has already provided a prompt to the user at least once in box 330. If computer 105 has not yet provided a prompt, process 300 proceeds to box 330. If computer 105 has already provided a prompt, process 300 ends.

[0064] In box 330, computer 105 outputs a message prompting the user to put on AR glasses 115 (if computer 105 recognizes the user as a pre-stored user in box 315) or outputs a message prompting the user to create a profile (if computer 105 does not recognize the user in box 315). After box 330, process 300 returns to decision box 320.

[0065] In box 335, computer 105 initiates AR display via AR glasses 115, in other words, sets AR glasses 115 as a possible destination for a portion of the output content 200.

[0066] Next, in box 340, computer 105 loads a restricted data policy from the user's profile identified in box 315. Computer 105 then makes this restricted data policy the active policy for the display of the currently managed content 200.

[0067] Next, in decision box 345, computer 105 identifies whether any portion of content 200 includes restricted data, as described above. In response to identifying a portion of content 200 as including restricted data, process 300 proceeds to box 350. In response to identifying content 200 as not including restricted data, process 300 proceeds to box 355.

[0068] In frame 350, computer 105 displays the first portion 205 using AR glasses 115 and prevents the first portion 205 from being displayed on display screen 110, as described above. After frame 350, process 300 proceeds to frame 355.

[0069] In box 355, computer 105 displays a second portion 210 (e.g., the remainder of content 200) on display screen 110, as described above. If computer 105 does not identify restricted data in content 200 in decision box 345, then the second portion 210 is the entire content 200.

[0070] Next, in decision box 360, computer 105 determines whether it has received input from the user to move a portion of content 200 between display screen 110 and AR glasses 115, as described above. In response to input from the user, process 300 proceeds to box 365. If no input is received, process 300 proceeds to decision box 370.

[0071] In box 365, computer 105 displays a portion of the content 200 indicated by input using AR glasses 115 and prevents said portion from being displayed on display screen 110, or displays said portion on display screen 110 instead of AR glasses 115, as indicated by input, as described above. After box 365, process 300 proceeds to decision box 370.

[0072] In decision box 370, computer 105 determines whether the length of time the user has used AR glasses 115 exceeds a threshold, as described above. If the length of time exceeds the threshold, process 300 proceeds to box 375. If the length of time is less than the threshold, process 300 proceeds to decision box 380.

[0073] In box 375, computer 105 prompts the user to remove AR glasses 115, as described above. After box 375, process 300 proceeds to decision box 380.

[0074] In decision box 380, computer 105 determines whether the user has removed AR glasses 115, as described above. In response to determining that the user has removed AR glasses 115, process 300 proceeds to box 385. In response to determining that the user is still wearing AR glasses 115, process 300 proceeds to decision box 390.

[0075] In box 385, computer 105 stops displaying the first portion 205 of content 200 on AR glasses 115, displays the first portion 205 on display screen 110, and locks display screen 110, as described above. After box 385, process 300 proceeds to decision box 390.

[0076] In decision box 390, computer 105 determines whether personal device 130 is no longer paired with computer 105 or has been turned off. If personal device 130 is still connected to computer 105, process 300 returns to box 310. If personal device 130 is no longer paired with computer 105 or has been turned off, process 300 ends.

[0077] Generally speaking, the described computing system and / or device may employ any of a variety of computer operating systems, including but not limited to the following versions and / or types: Ford Applications; AppLink / Smart Device Link middleware; Microsoft Operating system; Microsoft Operating system; Unix operating system (e.g., released by Oracle Corporation of Redwood Coast, California). Operating systems: AIX UNIX (published by International Business Machines, Inc., Armonk, New York); Linux; Mac OSX and iOS (published by Apple Inc., Inc., Cupertino, California); BlackBerry (published by BlackBerry Ltd., Inc., Waterloo, Canada); Android (developed by Google and the Open Handset Alliance); or provided by QNX Software Systems. CAR infotainment platform. Examples of computing devices include, but are not limited to, onboard computers, computer workstations, servers, desktop computers, laptops, mobile computers or handheld computers, or other computing systems and / or devices.

[0078] Computing devices typically include computer-executable instructions, which can be executed by one or more computing devices such as those listed above. Computer-executable instructions can be compiled or interpreted from computer programs created using a variety of programming languages ​​and / or technologies, which, individually or in combination, include, but are not limited to, Java. TM Languages ​​such as C, C++, Matlab, Simulink, Stateflow, Visual Basic, JavaScript, Python, Perl, and HTML are used. Some of these applications can be compiled and executed on virtual machines such as the Java Virtual Machine and the Dalvik Virtual Machine. Generally, a processor (e.g., a microprocessor) receives instructions (e.g., from memory, computer-readable media, etc.) and executes those instructions to perform one or more processes, including one or more processes described herein. Such instructions and other data can be stored and transferred using a variety of computer-readable media. Files in a computing device are typically collections of data stored on computer-readable media such as storage media, random access memory, etc.

[0079] Computer-readable media (also known as processor-readable media) include any non-transitory (e.g., tangible) medium that contributes to providing data (e.g., instructions) that can be read by a computer (e.g., by the computer's processor). Such media can take many forms, including but not limited to non-volatile and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, and internals that constitute a system bus coupled to the computer's processor. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, FLASH-EEPROM, any other memory chip or magnetic tape, or any other medium from which a computer can read.

[0080] The databases, data repositories, or other data stores described herein can include various mechanisms for storing, accessing / retrieving various types of data, including hierarchical databases, file sets in file systems, application databases in proprietary formats, relational database management systems (RDBMS), NoSQL databases, graph databases (GDB), and so on. Each such data store is typically contained within a computing device employing a computer operating system such as those mentioned above, and is accessed via a network in any one or more of various ways. File systems can be accessed from the computer operating system and can include files stored in various formats. In addition to languages ​​used to create, store, edit, and execute the stored programs (such as PL / SQL as described above), RDBMS typically employs Structured Query Language (SQL).

[0081] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) and stored on computer-readable media (e.g., disks, storage, etc.) associated therewith. Computer program products may include such instructions stored on computer-readable media for performing the functions described herein.

[0082] In the accompanying drawings, the same reference numerals indicate the same elements. Furthermore, some or all of these elements may be changed. Regarding the media, processes, systems, methods, inspirations, etc., described herein, it should be understood that although the steps of such processes, etc., are described as occurring in a certain ordered order, such processes can be practiced by performing the steps in a different order than that described herein. It should further be understood that some steps may be performed simultaneously, other steps may be added, or some steps described herein may be omitted. The operations, systems, and methods described herein should always be implemented and / or performed in accordance with applicable owner / user manuals and / or safety guidelines.

[0083] This disclosure has been described in an illustrative manner, and it should be understood that the terminology used is intended to describe the nature of the words, not to be restrictive. The adjectives “first” and “second” are used throughout this document as identifiers and are not intended to indicate importance, order, or quantity. The use of phrases such as “in response to,” “when… is determined,” and “when… is detected” indicates a causal relationship, not just a temporal one. In light of the foregoing teachings, many modifications and variations of this disclosure are possible, and this disclosure may be practiced in ways other than those specifically described.

[0084] According to the present invention, a computer is provided having a processor and a memory, the memory storing instructions executable by the processor to: determine, based on image data of a user depicting a display screen, that the user is wearing augmented reality glasses, the display screen being separate from the augmented reality glasses; and, in response to determining that the user is wearing the augmented reality glasses and in response to identifying a portion of content to be displayed on the display screen as including restricted data, display the portion with the augmented reality glasses and prevent the portion from being displayed on the display screen.

[0085] According to an embodiment, the instructions further include instructions for performing the following: identifying the user based on the image data, and in response to determining that the user is wearing the augmented reality glasses and identifying the user as a pre-stored user, displaying the portion with the augmented reality glasses, and preventing the portion from being displayed on the display screen.

[0086] According to an embodiment, the instructions also include instructions for receiving the image data from the camera.

[0087] According to an embodiment, the vehicle occupied by the user includes the camera.

[0088] According to an embodiment, the personal device being used by the user includes the camera and the display screen.

[0089] According to an embodiment, the personal device being used by the user includes the display screen.

[0090] According to an embodiment, the instructions also include instructions for pairing with the personal device and with the augmented reality glasses.

[0091] According to an embodiment, the instructions further include instructions for performing the following: in response to identifying a second portion of the content as not including restricted data, displaying the second portion on the display screen.

[0092] According to an embodiment, the instructions further include instructions for performing the following: in response to determining that the user is not wearing the augmented reality glasses, outputting a message prompting the user to put on the augmented reality glasses.

[0093] According to an embodiment, the instructions further include instructions for performing the following: locking the display screen in response to determining that the user has removed the augmented reality glasses.

[0094] According to an embodiment, the instructions further include instructions for performing the following: in response to input from the user, displaying a second portion of the content using the augmented reality glasses and preventing the second portion from being displayed on the display screen.

[0095] According to an embodiment, the instructions further include instructions for performing the following: identifying the portion of the content as including restricted data based on the fact that the portion of the content includes the user's personally identifiable information.

[0096] According to an embodiment, the instructions further include instructions for performing the following: identifying the portion as containing restricted data based on a data encryption tag associated with the portion of the content.

[0097] According to an embodiment, the instructions further include instructions for performing the following: identifying the portion of the content as including restricted data based on the fact that the portion includes age-restricted material.

[0098] According to the present invention, a method includes: determining, based on image data of a user depicting a display screen, that the user is wearing augmented reality glasses, the display screen being separate from the augmented reality glasses; and, in response to determining that the user is wearing the augmented reality glasses and in response to identifying a portion of content to be displayed on the display screen as including restricted data, displaying the portion with the augmented reality glasses and omitting the portion from the display screen.

[0099] In one aspect of the invention, the method includes: identifying the user based on the image data; and in response to determining that the user is wearing the augmented reality glasses and identifying the user as a pre-stored user, displaying the portion with the augmented reality glasses and preventing the portion from being displayed on the display screen.

[0100] In one aspect of the invention, the method includes receiving the image data from a camera.

[0101] In one aspect of the invention, the vehicle occupied by the user includes the camera.

[0102] In one aspect of the invention, the personal device being used by the user includes the camera and the display screen.

[0103] In one aspect of the invention, the method includes: displaying the second portion on the display screen in response to identifying a second portion of the content as not including restricted data.

Claims

1. A method comprising: determining, based on image data depicting a user of a display screen, that the user is wearing augmented reality glasses, the display screen being separate from the augmented reality glasses; and in response to determining that the user is wearing the augmented reality glasses and in response to identifying a portion of content for display by the display screen as including restricted data, displaying the portion with the augmented reality glasses and preventing the portion from being displayed on the display screen.

2. The method of claim 1 further comprising: identifying the user based on the image data; and in response to determining that the user is wearing the augmented reality glasses and identifying the user as a pre-stored user, displaying the portion with the augmented reality glasses and preventing the portion from being displayed on the display screen.

3. The method of claim 1 further comprising receiving the image data from a camera.

4. The method of claim 3 wherein a vehicle occupied by the user includes the camera.

5. The method of claim 3 wherein a personal device being used by the user includes the camera and the display screen.

6. The method of claim 1 wherein a personal device being used by the user includes the display screen.

7. The method of claim 6 further comprising pairing with the personal device and with the augmented reality glasses.

8. The method of claim 1 further comprising displaying a second portion of the content on the display screen in response to identifying the second portion as not including restricted data.

9. The method of claim 1 further comprising outputting a message prompting the user to put on the augmented reality glasses in response to determining that the user is not wearing the augmented reality glasses.

10. The method of claim 1 further comprising locking the display screen in response to determining that the user removed the augmented reality glasses.

11. The method of claim 1 further comprising displaying a second portion of the content with the augmented reality glasses and preventing the second portion from being displayed on the display screen in response to input from the user.

12. The method of claim 1 further comprising identifying the portion of the content as including restricted data based on the portion including personally identifiable information of the user.

13. The method of claim 1 further comprising identifying the portion of the content as including restricted data based on a data encryption flag associated with the portion.

14. The method of claim 1 further comprising identifying the portion of the content as including restricted data based on the portion including age-restricted material.

15. A computer comprising: a processor; and a memory storing instructions executable by the processor to perform the method of one of claims 1-14.

Images