Synchronized virtual reality notifications

Abstract

In a method of synchronizing notifications created in a VR environment to present the notifications in a physical environment at another time, notifications can be mapped across all relevant environments and can be associated with conditions that, when triggered, cause the notifications to be presented to a user. A computer-implemented method can include receiving, at a computing device in a first environment at a first time, a natural language request; parsing the natural language request into semantic components including a prompt component and a content component; determining one or more conditions based on the prompt component and the content component; mapping the semantic components to a plurality of environments; and if the one or more conditions are satisfied at a second time later than the first time, performing, by the computing device, a notification representing the prompt component and the content component in a second environment of the plurality of environments.

Description

Technical Field

[0001] This invention generally relates to the field of virtual reality, and more particularly to synchronous notification across virtual reality and physical environments. Background Technology

[0002] Interactions within virtual reality (VR) environments can be achieved using head-mounted displays and other interface devices, and can be used in a wide array of applications. Users equipped with wearable VR devices can be visually immersed in VR worlds, fully virtual VR environments, or a combination of virtual and physical environments. VR devices and environments are used for work, shopping, gaming, virtual travel, and other forms of entertainment. When accessing a specific VR environment using a VR device, the user is partially or completely disconnected from the physical world. Moreover, in most cases, the user is completely disconnected from other VR environments that the user might access at other times. Various VR features may be available or unavailable when the user navigates within a VR environment. For example, a user might perform a task in one VR environment that cannot be performed in another. This is a drawback for users who wish to perform tasks in one environment and translate those tasks into actions or events in another environment. Currently, another drawback is that users may have various ways to create notifications (e.g., prompts) specific to their current environment (e.g., virtual or physical), but users cannot create or modify notifications that are valid in other environments that the user can access or interact with at another time. Disjointed VR environments pose a challenge to users who want to synchronize multiple VR environments with tasks and events created within a single VR environment. Summary of the Invention

[0003] Embodiments of the present invention disclose a computer-implemented method, computer program product, and system for synchronizing notifications across virtual reality and physical environments. The computer-implemented method for synchronizing notifications between virtual reality and physical environments may include one or more processors configured to: receive a natural language request at a computing device operating within a first environment of multiple environments at a first time, wherein each of the multiple environments includes one or more parameters and the multiple environments are accessible by a user associated with the computing device; parse the natural language request into semantic components including cue components and content components; determine one or more conditions based on the cue components and the content components; map the semantic components to the multiple environments; and if the one or more conditions are met at a second time later than the first time, execute a notification by the computing device indicating the cue components and content components within the second environment of the multiple environments.

[0004] As a solution, synchronizing multiple VR environments allows a prompt or notification created in one environment to be triggered in another VR environment when certain conditions are met, offering advantages over challenges faced in existing technologies. These advantages allow users to immerse themselves in the entirety of a VR scene, where they can create events or notifications in one VR environment, move to another, and have those events or notifications applied there. Because events or notifications are synchronized across all applicable VR environments, the conditions associated with the event or notification can be triggered in the appropriate VR environment, where the user must take a related action or receive a set of associated information determined when the event or notification was created.

[0005] As a further advantage, the computer-implemented method may also include converting the natural language request into text data using semantic analysis. The computer-implemented method may also preferably include identifying cue components in the natural language request as device commands instructing the computing device to generate a notification. As another advantage, the computer-implemented method may also include identifying content components in the natural language request as user commands instructing the user to perform a task. Preferably, one or more conditions may include one or more of the following: geographical conditions corresponding to a location, activity conditions corresponding to a user event, and user conditions determined based on the content components. As an advantage, the first environment may be a virtual reality environment, and the second environment may be a physical environment, or vice versa. Preferably, the notification may be one or more of auditory and visual cues representing content components and is presented to the user via VR goggles communicatively coupled to the computing device. Attached Figure Description

[0006] Figure 1 A functional block diagram is depicted according to an embodiment of the present invention, which illustrates a distributed data processing environment for synchronizing notifications across virtual reality and physical environments;

[0007] Figure 2 A flowchart is depicted for a system for synchronizing notifications across virtual reality and physical environments according to an embodiment of the present invention;

[0008] Figure 3 A flowchart illustrating a method for synchronizing notifications across virtual reality and physical environments according to embodiments of the present invention is shown; and

[0009] Figure 4 The present invention is described in the form of an embodiment. Figure 1 A block diagram of computing devices in a distributed data processing environment. Detailed Implementation

[0010] The embodiments described herein provide computer-implemented methods, computer systems, and computer program products through which users can create cues or notifications in the physical world or in multiple VR environments (e.g., working, shopping, playing games, or other forms of human activity). Embodiments of the invention recognize that a cue entered in one environment may be most useful when presented in another environment (whether physical or virtual). The cues can be synchronized between two or more environments, including the physical world and the VR world. The cues can be presented based on conditions specified at the time the cues are created or at a later time. Depending on the conditions specified as part of the cues' attributes, the cues can be presented to the user or other users during their activities in any environment.

[0011] When using VR devices to access a specific VR environment, the user is partially or completely disconnected from the physical world, and in most cases, completely disconnected from other VR environments the user may use at different times. Users can employ various methods to create notifications (e.g., prompts) specific to their current environment (e.g., a physical or virtual environment), but they may not be able to create or modify notifications that are valid in environments accessed at different times. For example, at one point, a user might be using a VR device in a VR environment where they create a notification to be alerted to perform a task when one or more conditions are met (e.g., at a specific time, in a specific location, while observing a specific item). However, the user may not be able to create that notification in another environment because they are not in that environment or cannot access it. Therefore, a user-created notification in one VR environment can be presented to users in other environments when one or more conditions established when the notification is created are met.

[0012] In an embodiment, the computer-implemented method may include one or more processors configured to receive user input via a user interface of a user device associated with the user, wherein the user input may be processed to create or modify a notification (e.g., a prompt) in any of the physical or virtual environments in which the user device exists. The notification (e.g., the prompt) may then be synchronized across all environments and applied according to conditions specified or determined at or after the notification is created.

[0013] Synchronizing multiple VR environments allows a prompt or notification created in one environment to be triggered in another VR environment when certain conditions are met, offering advantages over challenges faced in existing technologies. These advantages allow users to immerse themselves across the entire scope of a VR scene, where a user can create an event or notification in one VR environment, move to another VR environment, and have the event or notification applied there as well. Because the event or notification is synchronized across all applicable VR environments, the conditions associated with the event or notification can be triggered in the appropriate VR environment where the user is advised to take an associated action or receive a set of associated information determined at the time the event or notification was created.

[0014] In one embodiment, a computer-implemented method may include one or more processors configured to detect a user device associated with a user interacting within a VR environment. When interacting within the VR environment, a user interface may be generated and configured to receive user input to create one or more notifications (e.g., prompts) based on one or more selected conditions within the VR environment. The one or more processors may be configured to apply the notification based on conditions specified or determined at the time the notification is created, while the user is accessing a different VR environment, or at a later time in the physical environment.

[0015] In one embodiment, a computer-implemented method may include one or more processors configured to receive user input to create one or more notifications (e.g., prompts) based on one or more selected conditions within a physical environment, wherein the notification may be presented to be detected by the user when a user is detected within one or more specific VR environments and the one or more selected conditions are met in the one or more specific VR environments.

[0016] In one embodiment, user input may correspond to an instruction to create a notification for one user or other users to receive. In other words, one or more processors may be configured to receive user input from a first user to create one or more notifications, which are then presented to a second user when a second user is detected in a specific VR environment or physical environment, provided one or more conditions are met. The user input for creating the notification can be performed via text input, a graphical user interface (GUI), speech-to-text, text-to-speech, or any other user data input method known to those skilled in the art.

[0017] In one embodiment, one or more processors may be configured to consider multiple factors when creating a notification and associated specified conditions to trigger it. For example, when mapping or synchronizing a notification and associated conditions to one or more environments, environmental location (e.g., VR or physical), geofence areas, time, contextual conditions, etc., may be considered. Other VR and physical environment parameters may be considered or embedded within one or more environments to determine whether to present a notification to the intended user. In one embodiment, notification parameters (e.g., user location, user behavior, and / or timing information) may be continuously considered and compared with the notification parameters to determine whether the conditions for presenting a notification to the intended user are met. In one embodiment, notification parameters may also include the location and / or behavior of other users (e.g., users other than the user responsible for creating the notification) when determining when to present a notification to the intended user in an environment (e.g., a VR environment and / or a physical environment).

[0018] By considering multiple factors and associating them with specified conditions to trigger a notification when it is created, the embodiments described herein allow for numerous customizations to be applied to each notification. Furthermore, it is advantageous to synchronize notifications across each applicable VR environment, enabling users to navigate freely throughout their entire domain while still receiving appropriate notifications once the conditions are met.

[0019] In one embodiment, one or more processors can be configured to receive input from a user to create a notification by receiving parameter input corresponding to conditions used to present the notification. For example, conditions for presenting the notification may include determining whether the user is in an environment corresponding to a store type or a specific geographic location, and then presenting the notification to the user. Another example of conditions for presenting the notification may include determining whether a certain amount of time has elapsed, and then presenting the notification to the user. Multiple conditions can be established to trigger the presentation of a notification to the user.

[0020] In one embodiment, the notification can be any form that is perceptible to the user. For example, the notification may include auditory cues, visual cues, haptic cues, message notifications, or any combination thereof. The notification may include specifications corresponding to multiple actions to be performed by one or more user devices associated with the user. Furthermore, the notification may include specifications corresponding to multiple actions to be performed by one or more user devices not associated with the user (e.g., a second user device associated with a first user device).

[0021] In one embodiment, a first user can set conditional notifications on a first user device, which, when the condition is met, can include a second user device presenting the notification to a second user. For example, a notification set by the first user can trigger a notification to be presented to the second user by the second user device for making a phone call or sending a text message. The notification may also include parameters or instructions regarding the environment in which the notification is presented. For example, if the user or another user is in a specific environment determined by detecting the expected user's geographic location, the user can provide conditions to present the notification only to that user or another user. Notification parameters may also include information about which specific users should receive the notification based on their permission, in which environment the notification is received, and in which timing parameters the notification should be presented.

[0022] In one embodiment, a computer-implemented method may include one or more processors configured to receive user input to create one or more notifications (e.g., prompts) based on one or more selected conditions when not in a physical environment. The notifications may also be presented when the user is not detected or when detected within a specific VR environment, and when one or more selected conditions are met both outside and within that specific VR environment. Multiple environments may each include one or more parameters, and multiple environments may be accessible to a user associated with a computing device.

[0023] In one embodiment, a computer-implemented method may include one or more processors configured to synchronize (e.g., map) one or more notifications within one or more environments (e.g., VR environment, physical environment) based on one or more conditions specified by a user or determined by one or more processors.

[0024] Once one or more notifications are synchronized or successfully mapped to one or more environments, one or more processors can be configured to determine whether a user is detected in that environment and, if one or more conditions associated with the notification are met, present the notification to the intended user.

[0025] In one embodiment, a user associated with one or more user devices (e.g., a smartphone, VR goggles, etc.) can be browsing products in a store while wearing one or more user devices (e.g., VR goggles coupled to a smartphone). As the user navigates the content in the store, the system can be configured to detect when the user purchases a first product and views a second product. One or more processors can be configured to analyze the user's shopping behavior (e.g., purchasing the first product and viewing the second product) and create a notification to send the user a prompt about the second product based on one or more conditions, which can be set by the user or determined by one or more processors based on one or more environmental parameters corresponding to the current user environment and / or the subsequent user environment.

[0026] In another embodiment, the computer-implemented method may include one or more processors configured to execute instructions on a computing device (e.g., a user device) to maintain a mapping between the physical and virtual environments by analyzing contextual parameters and conditions within both the physical and VR environments. For example, when a user is navigating a VR environment equipped with a VR user device, the VR user device may be configured to analyze VR content present in the VR environment to identify the user's virtual geographic location (e.g., virtual coordinates). Furthermore, when a user is navigating a VR environment equipped with a VR user device, the VR device may be configured to identify the user's virtual location, such as a shopping mall, train station, or other points of interest accessible to the user. Additionally, when a user is navigating a VR environment equipped with a VR user device, the VR device may be configured to identify or detect the presence of other users in the physical or VR environment.

[0027] In an example where a VR device is configured to detect the presence of other users in a VR environment, one or more processors can be configured to process the presence of other users to determine if conditions trigger a cue. For example, while the user is in the first environment, a message "Remind me to wish Trayvon a happy birthday" might have already been created. Then, once the VR device detects Trayvon's presence in another environment using its sensors, one or more processors can be configured to present a notification to the user that forwards the message "Wishing Trayvon a happy birthday." This notification could be presented as an audio message played from a speaker connected to the VR device, as a text message displayed so the user can visually view it, or as a unique haptic notification indicating whether the cue has been viewed or received.

[0028] In one embodiment, one or more processors may be configured to identify virtual geofence areas using contextual analysis of VR content present in a VR environment. When performing contextual analysis of the VR content, the one or more processors may be configured to process data corresponding to time, weather, user activity, user responses, and environmental parameters such as geographic location, timing, etc. (e.g., VR environment parameters and / or physical environment parameters). When performing contextual analysis to generate a notification, multiple factors (e.g., time, weather, user activity, user responses, and / or environmental parameters) may be combined from the physical and VR environment. For example, a VR user device may be activated and used by a user to participate in a tour with one or more tour routes, wherein the VR user device is configured to execute instructions corresponding to a tour guide operator for a specific route. During a tour on a first route, the VR user device may determine its first location based on location data (e.g., the first photo stop on a route in a national park). Furthermore, the VR user device may be configured to determine weather conditions (e.g., visibility, rainfall, and / or temperature) based on weather data. Additionally, the VR user device may be configured to determine time data (e.g., time of day) and use the time data to determine what time of day is suitable for participating in the tour. Furthermore, the VR user device can determine that a selected time of day is unsuitable for participation in the tour and provide a first notification to be detected by the user, prompting them to detour from the first tour route to a second tour route. Multiple notifications can be provided to the user of the VR user device to be detected. For example, in addition to the first notification, the VR user device can be configured to provide a second notification offering information that may be relevant to the user. The second notification may include telling the user “Enjoy the viewing!” Additionally, notifications may include alternative messages for the user to receive and make decisions (e.g., detour to a different route or continue on the current route).

[0029] The embodiments of the present invention can be implemented in various forms, and exemplary implementation details will be discussed below with reference to the accompanying drawings.

[0030] Figure 1 A functional block diagram according to an embodiment of the invention is depicted, illustrating a distributed data processing environment, typically designated as 100, for synchronizing notifications across virtual reality and physical environments. The term "distributed" as used herein describes a computer system comprising multiple physically distinct devices that operate together as a single computer system.

[0031] Figure 1 The illustrations provided are of only one embodiment of the invention and do not imply any limitation on the environments in which different embodiments may be implemented. Figure 1As shown, the distributed data processing environment 100 for synchronizing notifications across virtual reality and physical environments includes a network 110 configured to facilitate communication between a database 124, a server 125, a user device 130, and a virtual reality headset 132.

[0032] Network 110 operates as a computing network, which may be, for example, a local area network (LAN), a wide area network (WAN), or a combination of both, and may include wired, wireless, or fiber optic connections. Typically, network 110 can be any combination of connections and protocols supporting communication between database 124, server 125, user equipment 130, and virtual reality headset 132. It should also be understood that in some embodiments, network 110 is optional, and the distributed data processing environment 100 for synchronizing notifications across virtual reality and physical environments may operate as a standalone system, wherein in other embodiments, network 110 may be configured to enable user equipment 130 and / or virtual reality headset 132 to share a federated database using network 110.

[0033] User equipment 130 may be an electronic device configured to accompany a user. User equipment 130 may be a personal electronic device, such as a mobile communication device, smartphone, tablet computer, personal digital assistant, smart wearable device, personal laptop computer, desktop computer, or any other electronic device configured for user interaction and to collect user information to generate a user profile. In the depicted embodiment, user equipment 130 includes a user interface 122 and one or more sensors (not shown). User equipment 130 may include, for example... Figure 4 The components are described in further detail below.

[0034] User interface 122 serves as a local user interface on user device 130, through which one or more users of user device 130 interact with user device 130. In some embodiments, user interface 122 is a local application interface to a program (e.g., software configured to perform the steps of the invention described herein) on user device 130 or virtual reality headset 132. In some embodiments, user interface 122 is a graphical user interface (GUI), web user interface (WUI), and / or voice user interface (VUI), which can display (i.e., visually), present (i.e., audibly), and / or enable a user to input or receive information (i.e., graphics, text, and / or sound) for or from a program via network 110. In one embodiment, user interface 122 enables a user to send and receive data (i.e., send to and receive from a program via network 110, respectively). In one embodiment, user interface 122 enables a user to choose to join a program, input user-related data, and receive prompts to complete a task or activity.

[0035] The virtual reality headset 132 can be an electronic device configured to provide access to a VR environment. The electronic device may include wireless sensors, software, actuators, and computer equipment. The virtual reality headset 132 can be controlled via network 110 from a remote control system or via a local control system, or a combination of both. Furthermore, the virtual reality headset 132 can be configured to be controlled via a software application installed and executed by the virtual reality headset 132 or user equipment 130. When connected to a network, the virtual reality headset 132 can communicate usage data and other types of data corresponding to the device itself or other devices connected via network 110, where the data can provide useful insights within the scope of the designed application. The virtual reality headset 132 may be configured with a processor, memory, and peripheral devices (not shown) to receive and process data. The virtual reality headset 132 may include, for example, Figure 4 The components are described in further detail below.

[0036] In some embodiments, a user may wear special gloves and / or utilize a handheld controller in conjunction with a virtual reality headset 132 to perform operations in a VR environment. Optionally, one or more wearable sensors may be used to obtain various data about the user while the user operates in the virtual environment. This data may include, but is not limited to, location data, biometric data, and / or ambient environmental data.

[0037] Database 124 can serve as a repository for data associated with server 125, user device 130, virtual reality headset 132, and other data transmitted within network 110. A database is an organized collection of data. Database 124 can be implemented using any type of storage device capable of storing data and configuration files that can be accessed and utilized by server 125, user device 130, and / or virtual reality headset 132, such as a database server, hard drive, or flash memory. In one embodiment, database 124 can be accessed by server 125, user device 130, and / or virtual reality headset 132 to store data associated with user device 130 or virtual reality headset 132. In another embodiment, database 124 can be accessed by user device 130 or virtual reality headset 132 to access data as described herein. In one embodiment, database 124 can reside independently of network 110. In another embodiment, database 124 can reside elsewhere in the distributed data processing environment 100, provided that database 124 has access to network 110.

[0038] In the described embodiments, server 125 may include a program (i.e., software) configured to perform the steps of the invention described herein. In some embodiments, server 125 may be one or more standalone computing devices, one or more management servers, one or more web servers, one or more mobile computing devices, or one or more other electronic devices or computing systems capable of receiving, transmitting, and processing data. In some embodiments, server 125 may be a laptop computer, tablet computer, netbook computer, personal computer (PC), desktop computer, smartphone, or any programmable electronic device capable of communicating with user equipment 130 and virtual reality headset 132 via network 110. In other embodiments, server 125 represents a server computing system utilizing multiple computers as server systems, such as a cloud computing environment. In other embodiments, server 125 represents a computing system utilizing cluster computers and components (e.g., database server computers, application server computers, etc.) that act as a single seamless resource pool when accessed within a distributed data processing environment 100. Server 125 may include, for example, Figure 4 The components are described in further detail below.

[0039] Figure 2 A flowchart is depicted for a system 200 for synchronizing notifications across virtual reality and physical environments according to an embodiment of the present invention.

[0040] In an embodiment, system 200 may include one or more processors configured to receive user input 202 via a user interface of a user device (e.g., smartphone 230A) associated with a user, wherein the user input 202 may be processed to create or modify a notification 203 (e.g., a prompt) in either a virtual environment (e.g., virtual kitchen 240) or a physical environment (e.g., grocery store 250) in which the user device (e.g., smartphone 230A) is present. The notification 203 may then be synchronized across all environments (e.g., virtual kitchen 240 and grocery store 250) and applied according to conditions specified or determined when or after the user input 202 is used to create the notification 203.

[0041] In one embodiment, system 200 may include one or more processors configured to detect one or more user devices (e.g., smartphone 230a and VR glasses 230b) associated with a user interacting within a VR environment. When interacting within the VR environment, a user interface may be generated or made available, and the user interface may be configured to receive user input 202 to create one or more notifications 203 based on one or more selected conditions within the VR environment. One or more processors may be configured to apply the notification 203 according to conditions specified or determined in the physical environment, either when receiving and processing user input 202 to create the notification 203, or later when the user is accessing a different VR environment. Once the notification 203 is applied, one or more processors may be configured to present the notification 203 to be perceived by the user based on the conditions associated with the notification 203.

[0042] VR glasses 230B can be an electronic computing device that provides virtual reality capabilities. It may include a virtual reality headset, a mobile computing device (e.g., a smartphone, tablet, or other suitable computing device), a laptop computer, a desktop computer, and / or a wearable computing device. In an embodiment, a user uses VR glasses 230b to navigate a virtual environment (world) to perform virtual actions associated with tasks corresponding to similar tasks in the physical world. During task performance, various biometric and / or environmental data are acquired. This data drives product evaluation in the virtual environment and enables estimation of product performance in a similar physical environment.

[0043] In one embodiment, system 200 may include one or more processors configured to receive user input 202 to create one or more notifications 203 based on one or more selected conditions within a physical environment, wherein the notifications 203 may be presented to be detected by the user when a user is detected in one or more specific environments and when one or more selected conditions are met in one or more specific environments.

[0044] In one embodiment, user input 202 may correspond to an instruction to create notification 203 for a user to receive or for another user to receive. In other words, one or more processors may be configured to receive user input 202 from a first user to create one or more notifications 203, which are then presented to the second user when a second user is detected in a specific VR environment or a specific physical environment, provided one or more conditions are met. Receiving user input 202 to create notification 203 may include input via text input, a graphical user interface (GUI), speech-to-text, text-to-speech, or any other user data input method known to those skilled in the art.

[0045] In one embodiment, system 200 may include one or more processors configured to consider multiple factors when creating notification 203 and associating it with specified conditions to trigger notification 203. For example, when mapping or synchronizing notification 203 and associated conditions to one or more environments (e.g., VR environments and / or physical environments), environmental location, geofence regions, time, context, etc., may be considered. Other VR environment parameters and physical environment parameters may be considered or embedded within one or more environments to determine whether to present the notification to the intended user. In an embodiment, notification parameters (e.g., user location, user behavior, and / or timing information) may be continuously considered and compared with the notification parameters to determine whether the conditions for presenting notification 203 to the intended user are met. In one embodiment, notification parameters may also include the location and / or behavior of other users (e.g., users other than the user responsible for creating the notification) when determining when to present the notification to the intended user in an environment (e.g., VR environment and / or physical environment).

[0046] In one embodiment, system 200 may include one or more processors configured to receive user input 202 from a user to create notification 203 by receiving parameter input corresponding to conditions for presenting notification 203. For example, conditions for presenting notification 203 may include determining whether the user is in an environment corresponding to a store type or a specific geographic location, and then presenting notification 203 to the user. Another example of conditions for presenting notification 203 may include determining whether a certain amount of time has elapsed, and then presenting notification 203 to the user. Multiple conditions may be established to trigger the presentation of notification 203 to the user.

[0047] In one embodiment, system 200 may include one or more processors configured to convert user input 202 from speech to text using information extraction techniques and semantic analysis to process user input 202. User input 202 may include instructions to perform a task, where the task may be to purchase a specific item 201. In one embodiment, item 201 may be a bunch of garlic cloves, where item 201 can be identified via VR glasses 230b as the user browses virtual kitchen 240. For example, if user input 202 includes the text “Don’t forget to buy garlic,” one or more processors may be configured to identify the action to be performed as “purchase,” the object to be purchased or acquired as “garlic,” and the location of the garlic purchase as “grocery store.” The action, object, and location determined by performing information extraction and semantic analysis may be stored in notification database 224 for further processing and transmission.

[0048] In one embodiment, system 200 may include one or more processors configured to transmit data corresponding to notification 203 to smartphone 230a for presentation to the user of smartphone 230a if conditions for presenting notification 203 are met. Furthermore, one or more processors may be configured to compare location data and other parameters with the data corresponding to notification 203 to determine whether and when notification 203 should be presented to the user. For example, if the conditions associated with notification 203 correspond to the user being within the vicinity of a physical environment corresponding to grocery store 250, one or more processors may be configured to present notification 203 to the user once smartphone 230a has determined that the user is within the vicinity of the physical environment corresponding to grocery store 250.

[0049] In one embodiment, notification 203 can be any form that is perceptible to the user. For example, notification 203 may include auditory cues, visual cues, tactile cues, message prompts, or any combination thereof. Notification 203 may include specifications corresponding to multiple actions to be performed by one or more user devices associated with the user (e.g., smartphone 230a, VR glasses 230b). Furthermore, notification 203 may include specifications corresponding to multiple actions to be performed by one or more user devices not associated with the user (e.g., a second user device associated with a first user device, a second user device not associated with a first user).

[0050] In one embodiment, a first user can provide user input 202 via a first user device (e.g., smartphone 230a, VR glasses 230b) to create a conditional notification 203, which, when the condition is met, may include one or more processors configured to present the notification 203 to a second user associated with a second user device (not shown). For example, user input 202 by the first user to create the notification 203 may trigger a message 203 to be presented to the second user by the second user device (not shown) for making a phone call or sending a text message. The notification may also be created to include parameters or instructions regarding which environment is used to present the notification. For example, if the user or another user is in a specific environment determined by detecting the geographic location of the expected user, the user can provide conditions to present the notification only to the user or another user. The notification parameters may also include timing parameters regarding which specific users should receive the notification based on which users' permission, in which environment the notification should be received, and when the notification should be presented.

[0051] In one embodiment, system 200 may include one or more processors configured to receive user input 202 to create one or more notifications 203 based on one or more selected conditions when not in a physical environment, wherein the notifications 203 may be presented to the user even when no user is detected in a particular VR environment and even when one or more selected conditions are met while not in a particular VR environment.

[0052] In one embodiment, system 200 may include one or more processors configured to synchronize (e.g., map) one or more notifications 203 with one or more environments (e.g., VR environments and / or physical environments) based on one or more conditions specified by the user or determined by one or more processors.

[0053] Once one or more notifications 203 are synchronized or successfully mapped to one or more environments, one or more processors can be configured to determine whether a user is detected in one or more environments, and to present one or more notifications 203 to the intended user if one or more conditions associated with one or more notifications 203 are met.

[0054] In one embodiment, a user associated with one or more user devices (e.g., smartphone 230a, VR glasses 230b) may be browsing products in a store while wearing VR glasses 230b coupled to smartphone 230a. While the user is navigating content in the store, system 200 may be configured to detect, via one or more user devices, a user who purchases a first product and views a second product. One or more processors may be configured to analyze the user's shopping behavior (e.g., purchasing the first product and viewing the second product) and create notification 203 to send a prompt about the second product to the user based on one or more conditions, which may be set by the user or determined by one or more processors based on one or more environmental parameters corresponding to the current user environment and / or the user environment later.

[0055] In another embodiment, system 200 may include one or more processors configured to execute instructions on a computing device (e.g., smartphone 230a, VR glasses 230b) to maintain a mapping between the physical and virtual environments by analyzing contextual parameters and conditions in both the physical and VR environments. For example, when a user is navigating a VR environment (e.g., virtual kitchen 240) equipped with a VR user device (e.g., VR glasses 230b), the VR user device may be configured to analyze VR content present in the VR environment to identify the user's virtual geographic location (e.g., virtual coordinates). Furthermore, when a user is navigating a VR environment (e.g., virtual kitchen 240) equipped with a VR user device (e.g., VR glasses 230b), the VR device (e.g., VR glasses 230b) may be configured to identify the user's virtual context location, such as a shopping mall, train station, or other points of interest accessible to the user. Additionally, when a user is navigating a VR environment equipped with a VR user device, the VR device may be configured to identify or detect the presence of other users in the physical or VR environment.

[0056] Figure 3 A flowchart is depicted of a computer-implemented method 300 for synchronizing notifications across virtual reality and physical environments according to an embodiment of the present invention.

[0057] In one embodiment, the computer-implemented method 300 may include receiving a natural language request 302 at a computing device configured to operate in a first environment within a plurality of environments, wherein at least one of the environments is a VR environment. For example, the natural language request may correspond to user input in the form of text data, audio input, or other user data intended as user input to the computing device. Audio input may include spoken speech that can be converted into audio data via one or more processors. The phenotype of the audio data may be analyzed and converted into text for further processing. In some embodiments, natural language processing may be performed within the... Figure 1 The natural language processing is performed on-board on user equipment (e.g., user equipment 130) and server 125. In other embodiments, all or some of the natural language processing may be performed on a remote computer.

[0058] In one embodiment, a natural language processing (NLP) system, such as a machine learning system, can be used to analyze audio data. The machine learning system can be used to further classify and categorize input data, including biometric data from sensors, image data, scene classification algorithms, object recognition and / or object classification, person recognition, natural language processing (NLP), sentiment analysis, and / or other classification processes. The machine learning system may include one or more neural networks, convolutional neural networks (CNNs), and / or other deep learning techniques. The machine learning system may include regression algorithms, classification algorithms, clustering techniques, anomaly detection techniques, Bayesian filtering, and / or other suitable techniques to analyze information obtained by one or more processors, thereby aiding in the classification of the information.

[0059] In one embodiment, the computer-implemented method 300 may include one or more processors configured to parse a natural language request 304 into semantic components that include cue components and content components. For example, parsing a natural language request 304 into semantic components may include using semantic analysis to convert the natural language request into text data. Furthermore, parsing 304 a natural language request may include identifying text corresponding to actions as cue components and text corresponding to objects and / or locations as content components.

[0060] In one embodiment, the computer-implemented method 300 may include one or more processors configured to recognize prompting components in a natural language request as device commands or actions instructing a computing device to generate a notification.

[0061] In one embodiment, the computer-implemented method 300 may include one or more processors configured to recognize content components in a natural language request as user commands instructing a user to perform a task.

[0062] In one embodiment, the computer-implemented method 300 may include one or more processors configured to determine one or more conditions 306 based on cue components and content components. In one embodiment, the one or more conditions may include one or more of geographical conditions corresponding to a location, activity conditions corresponding to a user event, and user conditions determined based on the content components. A user event may include a user activity corresponding to a natural human action performed at a specific time and / or location. For example, a user event may include a user walking into a grocery store, driving past a retail store, or jogging past a clothing store. User events may be determined by one or more processors by processing data collected by sensors communicating with a computing device associated with the user.

[0063] In one embodiment, the computer-implemented method 300 may include one or more processors configured to map semantic components 308 to multiple environments.

[0064] In one embodiment, the computer-implemented method 300 may include one or more processors configured to be executed by a computing device to notify 310 that if one or more conditions are met at a second time later than the first time, the notification represents a prompt component and a content component in a second environment of multiple environments.

[0065] In one embodiment, the first environment may include a virtual reality environment, and the second environment may include a physical environment.

[0066] In one embodiment, the notification may be one or more of auditory and visual cues representing content components.

[0067] Figure 4 An embodiment of the invention is depicted. Figure 1 A block diagram of computing devices 400 in a distributed data processing environment 100, such as server 125, user equipment 130, and / or virtual reality headset 132. It should be understood that... Figure 4 This is merely an illustration of one implementation and does not imply any limitation on the environment in which different embodiments may be implemented. Many modifications can be made to the described environment.

[0068] The computing device 400 includes a communication structure 402 that provides communication between a cache 416, a memory 406, a persistent storage device 408, a communication unit 410, and an input / output (I / O) interface 412. The communication structure 402 can be implemented using any architecture designed to transfer data and / or control information between processors (such as microprocessors, communication and network processors, etc.), system memory, peripheral devices, and any other hardware components within the system. For example, the communication structure 402 can be implemented using one or more buses or crossbar switches.

[0069] Memory 406 and persistent storage device 408 are computer-readable storage media. In this embodiment, memory 406 includes random access memory (RAM). Typically, memory 406 may include any suitable volatile or non-volatile computer-readable storage medium. Cache 416 is a fast memory that enhances the performance of computer processor 404 by storing recently accessed data and data from memory 406 that is about to be accessed.

[0070] Software and data 414 may be stored in persistent storage device 408 and memory 406 for execution and / or access by one or more of the respective computer processors 404 via cache 416. In one embodiment, persistent storage device 408 includes a magnetic hard disk drive. As an alternative to or supplement to a magnetic hard disk drive, persistent storage device 408 may include a solid-state drive, semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0071] The media used in persistent storage device 408 can also be removable. For example, a removable hard disk drive can be used for persistent storage device 408. Other examples include optical discs and disks, thumb drives and smart cards, which are inserted into the drive to transfer data to another computer-readable storage medium that is also part of persistent storage device 408.

[0072] In these examples, communication unit 410 provides communication with other data processing systems or devices. In these examples, communication unit 410 includes one or more network interface cards. Communication unit 410 can provide communication by using one or both of physical and wireless communication links. Software and data 414 can be downloaded to permanent storage device 408 via communication unit 410.

[0073] One or more I / O interfaces 412 allow data input and output to other devices that can be connected to database 124, server 125, user equipment 130, and / or virtual reality headset 132. For example, I / O interface 412 can provide connectivity to external devices 418 such as keyboards, keypads, touchscreens, and / or other suitable input devices. External devices 418 may also include portable computer-readable storage media, such as thumb drives, portable optical discs or disks, and memory cards. Software and data 414 used to implement embodiments of the invention can be stored on such portable computer-readable storage media and can be loaded onto persistent storage device 408 via one or more I / O interfaces 412. I / O interface 412 is also connected to display 420.

[0074] The display 420 provides a mechanism for displaying data to the user and can be, for example, a computer monitor.

[0075] The programs described herein are identified based on applications in which they are implemented in specific embodiments of the invention. However, it should be understood that any particular procedural terminology used herein is for convenience only, and therefore the invention should not be limited to use only in any particular application identified and / or implied by such terminology.

[0076] This invention can be a system, a computer-implemented method, and / or a computer program product. A computer program product may include one or more computer-readable storage media having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0077] Computer-readable storage media can be any tangible device capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0078] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0079] Computer-readable program instructions for performing the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Smalltalk, C++, etc.) and conventional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information from the computer-readable program instructions.

[0080] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0081] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0082] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a non-consecutive order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0084] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies on the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for synchronizing notifications across virtual reality and physical environments, the computer-implemented method comprising: A natural language request is received in a first-time manner by one or more processors at a computing device operating within a virtual reality environment running in multiple environments, each of which includes one or more parameters, and the multiple environments are accessible by a user associated with the computing device; The natural language request is parsed into semantic components, including prompting components and content components, by one or more processors; One or more conditions are determined by the one or more processors based on the prompting component and the content component; The semantic components are mapped to the plurality of environments by the one or more processors; as well as If one or more conditions are met at a second time later than the first time, a notification is executed by the one or more processors through the computing device, the notification representing the prompt component and the content component in the physical environment of the plurality of environments; as well as The mapping between the physical environment and the virtual reality environment is maintained by one or more processors by analyzing context parameters and conditions in the physical environment and the virtual reality environment.

2. The computer-implemented method according to claim 1 further includes: The natural language request is converted into text data by the one or more processors using semantic analysis.

3. The computer-implemented method according to any one of claims 1 to 2, further comprising: The one or more processors recognize the prompting component in the natural language request as a device command instructing the computing device to generate the notification.

4. The computer-implemented method according to any one of claims 1 to 2, further comprising: The one or more processors identify the content components in the natural language request as user commands instructing the user to perform a task.

5. The computer-implemented method according to any one of claims 1 to 2, wherein, The one or more conditions include one or more of the following: geographical conditions corresponding to location, activity conditions corresponding to user events, and user conditions determined based on the content components.

6. The computer-implemented method according to any one of claims 1 to 2, wherein, Maintaining the mapping between the physical environment and the virtual reality environment includes analyzing virtual reality content existing in the virtual reality environment to identify the user's virtual geographic location, the user's virtual environment location, or the presence of other users in the virtual reality environment or the physical environment.

7. The computer-implemented method according to any one of claims 1 to 2, wherein the notification is one or more of an auditory cue and a visual cue representing the content component, and is presented to the user via VR goggles communicatively coupled to the computing device.

8. A computer program product for synchronizing notifications across virtual reality and physical environments, the computer program product comprising: One or more computer-readable storage media and program instructions commonly stored on the one or more computer-readable storage media, the stored program instructions including: Program instructions for receiving natural language requests at a computing device operating within a virtual reality environment running in multiple environments in a first-time manner, wherein each of the multiple environments includes one or more parameters, and the multiple environments are accessible by a user associated with the computing device; Program instructions for parsing the natural language request into semantic components, including prompting components and content components; Program instructions for determining one or more conditions based on the prompting component and the content component; Program instructions for mapping the semantic components to the plurality of environments; and Program instructions for executing a notification by the computing device if one or more conditions are met at a second time later than the first time, the notification representing the prompt component and the content component in the physical environment of the plurality of environments; and Instructions for maintaining the mapping between the physical environment and the virtual reality environment by analyzing context parameters and conditions in the physical environment and the virtual reality environment.

9. The computer program product according to claim 8, further comprising: Program instructions for using semantic analysis to convert the natural language request into text data.

10. The computer program product according to any one of claims 8 to 9, further comprising: Program instructions for identifying the prompting component in the natural language request as a device command instructing the computing device to generate the notification.

11. The computer program product according to any one of claims 8 to 9, further comprising: Program instructions for recognizing the content components in the natural language request as user commands instructing the user to perform a task.

12. The computer program product according to any one of claims 8 to 9, wherein the one or more conditions include one or more of the following: geographical conditions corresponding to a location, activity conditions corresponding to a user event, and user conditions determined based on the content components.

13. The computer program product according to any one of claims 8 to 9, wherein maintaining the mapping between the physical environment and the virtual reality environment includes analyzing virtual reality content existing in the virtual reality environment to identify the user's virtual geographic location, the user's virtual environment location, or the presence of other users in the virtual reality environment or the physical environment.

14. The computer program product according to any one of claims 8 to 9, wherein the notification is one or more of an auditory and visual cue representing the content component, and is presented to the user via VR goggles communicatively coupled to the computing device.

15. A computer system for synchronizing notifications across virtual reality and physical environments, the computer system comprising: One or more computer processors; One or more computer-readable storage media; Program instructions jointly stored on the one or more computer-readable storage media for execution by at least one of the one or more computer processors, the stored program instructions including: Program instructions for receiving natural language requests at a computing device that operates in a first environment within a plurality of environments at a first time, wherein each of the plurality of environments includes one or more parameters, and the plurality of environments are accessible by a user associated with the computing device; Program instructions for parsing the natural language request into semantic components, including prompting components and content components; Program instructions for determining one or more conditions based on the prompting component and the content component; Program instructions for mapping the semantic components to the plurality of environments; and Program instructions for executing a notification by the computing device if one or more conditions are met at a second time later than the first time, the notification representing the prompt component and the content component in a second environment of the plurality of environments; Instructions for maintaining the mapping between the physical environment and the virtual reality environment by analyzing context parameters and conditions in the physical environment and the virtual reality environment.

16. The computer system according to claim 15, further comprising: Program instructions for using semantic analysis to convert the natural language request into text data.

17. The computer system according to any one of claims 15 to 16, further comprising: Program instructions for recognizing the prompting components in the natural language request as device commands instructing the computing device to generate the notification; as well as Program instructions for recognizing the content components in the natural language request as user commands instructing the user to perform a task.

18. The computer system according to any one of claims 15 to 16, wherein the one or more conditions include one or more of the following: geographical conditions corresponding to a location, activity conditions corresponding to a user event, and user conditions determined based on the content components.

19. The computer system according to any one of claims 15 to 16, wherein maintaining the mapping between the physical environment and the virtual reality environment includes analyzing virtual reality content existing in the virtual reality environment to identify the user's virtual geographic location, the user's virtual environment location, or the presence of other users in the virtual reality environment or the physical environment.

20. The computer system of any one of claims 15 to 16, wherein the notification is one or more of an auditory and visual cue representing the content component, and is presented to the user via VR goggles communicatively coupled to the computing device.

Images