System and method for augmented reality interaction with object using head-based wearable device

Abstract

Systems and methods for interacting with a physical object are provided. Technologies include receiving, via a head-based wearable device, data parameters associated with a user; receiving, via the head-based wearable device, a data parameter associated with the object; determining that the user is near the object; transmitting the user data parameter and the object data parameter to a processor configured to: identify at least one gaze vector object definition with the object based on the object data parameter; identifying, with the user, at least one gaze vector object matrix; determining a user contact state with the object; setting an execution value based on the user contact state; and it transmits the execution value to an output server.

Description

Background Art

[0001] Enhancing the user's interaction with the physical world around them through complex collaborative analysis of the user's biodata and real-world object data in an augmented reality environment is an area of ​​great interest. Some markets focus on reading brain waves to operate devices or communicate with voice. Devices that read brain waves and control devices are often referred to as brain-computer interfaces (BCI), mind-machine interfaces (MMI), brain-ports (DNI), synthetic telepathic interfaces (STI), or brain-machine fusion perception (BMI).

[0002] A BCI is a system in which messages or commands sent by a user to the outside world do not pass through the normal output pathways of the brain's peripheral nerves and muscles. For example, in an electroencephalogram (EEG)-based BCI, information is encoded in the EEG activity. BCIs provide users with an alternative way to interact with the world. Some early BCIs are described in Wolpaw, Jonathan R., N. Birbaumer, DJ McFarland, G. Pfurtscheller, and T. M. Vaughan, "Brain-computer interfaces for communication and control," Clinical Neurophysiology, 113 (2002) 767-791.

[0003] BCI measures electrophysiological signals from the user's brain and uses hardware and software to translate those signals into specific commands that mimic the user's intended actions.

[0004] In traditional systems, devices such as EEG headsets read brain activity in the form of electrical waves. These waves can be divided into six categories: delta, theta, alpha, beta, gamma, and mu. Typically, these waves are read in real time, interpreted, and then associated with predefined actions, resulting in a "one-to-one association," meaning that the user thinks "X," and the device either says "X" or performs "X."

[0005] There are devices that can convert a user's thoughts into directions (e.g., directions along an axis) or motions (e.g., motions of a user's body part, or motions of an external object), because the user imagines a motion or direction, such as "left", or "ball moves up", or "hand moves down", which produces an easily recognizable wave pattern for the EEG to read, and the EEG sends that pattern to proprietary software, which can then send the appropriate corresponding (e.g., "left") commands to simple objects such as a computer cursor or mouse pointer, as well as physical objects such as a wheelchair, car, or other device, or can use the thought as a component in the output of an information item. Examples include a user thinking "left", which causes something to move left (e.g., a prosthetic limb, a mouse pointer, etc.).

[0006] There are inherent challenges in associating brainwave patterns with spoken or performed actions in existing one-to-one association models. Existing thought-to-speech technology requires a computer chip to be surgically implanted in the user's brain. Existing thought-to-speech technology also has difficulty accurately identifying common / frequent "patterns" in brainwaves for both spoken and "non-spoken / thought / imagined" language. Often, these patterns may be the same for similar words: cat, rat, bat, mat, etc. Or similar phrases, for example, "I hate cats" and "irate bats." Even with the chip implanted, users can expect a success rate between 45% and 89%. For more on thought-to-speech, see "Machine Converts Thoughts to Speech in Real Time," December 21, 2009.

[0007] If an "over the counter" EEG machine is combined with existing forms of brainwave interpretation, the resolution of the device is often not precise enough to consistently identify useful intended thoughts. Not only is general interpretation of brainwaves and specific brainwave patterns very difficult due to their dynamic nature and high variability, but external electrical interference from commonly used devices such as cell phones and televisions makes this process even more difficult and the output even more inaccurate.

[0008] The challenges in the user thought-command category of thought-controlled devices are almost the same as those in the thought-speech category described above. The problem of correctly identifying non-verb commands greatly affects the selection of relevant actions.

[0009] There are also challenges with the user's thought-direction category. The one-to-one association of these non-verbal commands / thoughts with the direction of "something" is the most accurate and repeatable process for all thought-controlled devices to date. Up, down, left, and right are all consistent, repeatable patterns and are probably the most easily recognizable markers in the EEG brainwave data that is commonly collected. However, this category is limited because researchers and inventors have only used this logic to associate direction-based thoughts with the movement of objects (e.g., movement of a mouse pointer, movement of a computer cursor, operation of a wheelchair, driving a car, operating an exoskeleton, etc.), which maintains a one-to-one association of "imagined direction" with the direction actually executed, whether it is virtual through software or mechanical through devices and equipment.

[0010] This challenge / limitation leaves a gap in controlling devices that do not operate using directional thoughts. For example, controlling devices such as televisions, doors, appliances, adjustable beds, and even cell phones and tablet computers by using thoughts is not easily accomplished using traditional methods. The answer will lead one to operate in the thought-speech or thought-command mode, which is very unreliable and unpredictable. Due to the traditional cognitive limitation to a limited number of directional thoughts, it will not necessarily lead one to operate in the thought-direction mode.

[0011] Other difficulties arise when a user of a BCI head-based wearable device is near multiple real-world objects for potential interaction through augmented reality. Existing systems and methods generally have difficulty accurately identifying the user's intention to meaningfully contact a specific object in a group of objects, and making an intelligent decision about whether to allow the user to interact with the specific object via a head-based wearable device (e.g., passing commands to control the object, or receiving information content from the object). For example, a user who keeps a direct gaze on a real-world physical world object near the user (e.g., a storefront, a billboard, or a statue of a historical figure) will likely have a desire to interact with the object via a head-based wearable device. In contrast, a user who only stands in front of such an interactive object without maintaining a direct line of sight with the object (e.g., a user who is reading an email message on his mobile phone) may not have an actual intention or desire to interact with the specific object.

[0012] There is a need for improved systems and methods that use thought and other inputs to define individual actions, functions, or executions of non-tactile (e.g., without the touch input required for a keyboard or mouse) or thought-controlled devices more reliably and accurately. Additionally, there is a need for systems and methods that will accurately recognize user intent to meaningfully engage objects in the surrounding physical world and allow users to navigate and communicate complex inputs to information systems that interact with objects in a non-tactile manner (i.e., without speaking or touching). Summary of the invention

[0013] Disclosed embodiments include a method for interacting with a physical object using a head-based wearable device. The method includes receiving data parameters associated with a user via the head-based wearable device; receiving data parameters associated with an object via the head-based wearable device; and determining that the user is near the object. The method may further include transmitting the user data parameters and the object data parameters to a processor, wherein the processing unit is configured to identify at least one line of sight vector object definition using the object based on the object data parameters, identify at least one line of sight vector object matrix using the user, determine a user contact state with the object, set an execution value based on the user contact state, and transmit the execution value to an output server.

[0014] The disclosed embodiments also include a system for interacting with a physical object. The system may include a head-based wearable device, a memory for storing instructions, and at least one processor configured to execute a set of instructions to: receive data parameters associated with a user via the head-based wearable device; receive data parameters associated with an object via the head-based wearable device; and determine that the user is near the object. The processor may be further configured to identify at least one sight vector object definition using the object based on the object data parameters, identify at least one sight vector object matrix using the user, determine a user contact state with the object, set an execution value based on the user contact state, and transmit the execution value to an output server.

[0015] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings are not necessarily drawn to scale or exhaustive. Instead, emphasis is generally placed on illustrating the principles of the invention described herein. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several embodiments consistent with the present disclosure and, together with the detailed description, serve to explain the principles of the present disclosure. In the drawings:

[0017] Figure 1 A schematic diagram of an exemplary system 100 for interacting with real-world physical objects using a head-based wearable device is depicted.

[0018] Figure 2 An exemplary method 200 for interacting with real-world physical objects using a head-based wearable device based on a gaze vector object matrix is ​​described.

[0019] Figure 3 An exemplary device as part of a system for interacting with physical objects using a head-based wearable device is described.

[0020] Figure 4 An exemplary set of gaze vector object parameters within a gaze vector object matrix as part of a system for interacting with physical objects using a head-based wearable device is described.

[0021] Figure 5 An exemplary line-of-sight vector object matrix is ​​described, containing various types of access levels and associated sets of line-of-sight vector object definitions within each matrix. DETAILED DESCRIPTION

[0022] In the following detailed description, many specific details are set forth to provide a comprehensive understanding of the disclosed example embodiments. However, those skilled in the art will appreciate that the principles of the example embodiments can be implemented without each specific detail. Well-known methods, processes, and components are not described in detail to avoid obscuring the principles of the example embodiments. Unless explicitly stated, the example methods and processes described herein are not limited to a specific order or sequence, nor to a specific system configuration. In addition, some of the described embodiments or elements thereof may occur or execute simultaneously, at the same time point, or concurrently. Reference will now be made to the disclosed embodiments, examples of which are shown in the accompanying drawings.

[0023] As used herein, "network" shall refer to one or more networks of any type, including networks capable of being used with the data conversion and distribution system, such as any public and / or private network, including, for example, the Internet, an intranet or extranet, any wired or wireless network, or a combination thereof.

[0024] The systems and methods disclosed herein allow for detection of user contact with real-world objects based on comprehensive user-object data analysis via a BCI head-based wearable device. Such systems and methods further enable unique personalized user-object interactions (e.g., controlling or commanding an object or receiving audio feedback in a non-tactile manner) based on specifically defined user access layers. The disclosed systems and methods enable such interactions in an augmented reality environment using a head-based wearable device, solve technical problems that arise in user-object interactions, and provide technical solutions to existing problems in the field.

[0025] Figure 1 A schematic diagram of an exemplary system 100 for interacting with real-world physical objects using a head-based wearable device is described. In some embodiments, the system 100 may include a head-based wearable device 101, one or more processing units 111, one or more cloud input / output (I / O) servers 121, and at least one SVO database server 130.

[0026] The head-based wearable device 101 is a device configured to collect biosignals corresponding to a user. The head-based wearable device 101 may include, for example, a biosignal sensor 102, an inertial measurement unit (IMU) sensor 103, and an air pressure sensor 104. The device 101 may also include a memory component 105, a communication module 106, and a power supply component 107. The device 101 may also include a processor 108, a device controller 109, and an audio feedback module 110.

[0027] In some embodiments, the head-based wearable device 101 may be, for example, one or more earbuds. An example of a suitable earbud is described in U.S. Patent No. 10,275,027, "Apparatus, method, and system for defining actions, functions, or executions using imagined directions." In some embodiments, the device 101 may be augmented reality glasses. In some embodiments, the device 101 may be configured to fit substantially within the ear canal.

[0028] like Figure 1 As shown, the head-based wearable device 101 may include a biosignal sensor 102, which may be configured to receive one or more biosignals from a user. The biosignals used here may include, for example, gesture data, electromyography (EMG) data, or electroencephalography (EEG) data. In some embodiments, the biosignal sensor 102 may include a gesture sensor. In some embodiments, the biosignal sensor 102 may include an EMG sensor. In some embodiments, the biosignal sensor 102 may include an EEG sensor.

[0029] like Figure 1 As shown, device 101 may include an IMU sensor 103. An IMU is a collection of one or more sensors that capture different data types depending on the type of sensor in the IMU. For example, an IMU may include one or more accelerometers, gyroscopes, and / or magnetometers. For example, an accelerometer measures velocity and acceleration. A gyroscope can be used to measure rotation and rotation rate. A magnetometer captures data representing basic directions (direction heading). In some embodiments, IMU 103 can measure at least 3 degrees of freedom. IMU 103 may be configured to receive head position data from a user. In some embodiments, IMU 103 may be configured to collect angular momentum data associated with the user's head position. In some embodiments, IMU may be configured to collect force data associated with the user's head position.

[0030] like Figure 1 As shown, device 101 may include air pressure sensor 104. In some embodiments, sensor 104 may be configured to collect air pressure data from a user. In some embodiments, sensor 104 may be configured to receive air pressure data from an integrated pressure sensor within device 101. In some embodiments, sensor 104 may be configured to receive air pressure data from an external sensor connected to device 101. In some embodiments, a processing unit (e.g., processing unit 111) may analyze the data collected by air pressure sensor 104 to determine an altitude value of a user or object.

[0031] In some embodiments, the device 101 may include a memory component 105 provided as a non-transitory memory component configured to store software that causes a process component 108 coupled to the bio-signal sensor 102 to perform operations according to the software. In some embodiments, the memory component 105 may aggregate one or more bio-signals to a separate auxiliary device or processing unit 111. In some embodiments, the memory component 105 may transmit the one or more bio-signals to a separate auxiliary device, such as, but not limited to, a mobile device or a computer, via a communication component.

[0032] In some embodiments, device 101 may include a communication input / output (I / O) module 106. In some embodiments, communication module 106 may transmit data from device 101 to an external device, such as a cell phone or computer. In some embodiments, communication module 106 may receive data from an external source, such as processing unit 111.

[0033] In some embodiments, device 101 can include a power supply component 107. Component 107 can be configured to power one or more of bio-signal sensor 102, memory component 105, or communication I / O module 106.

[0034] In some embodiments, the device 101 may include a processor 108. In some embodiments, the processor 108 may be configured to process biosignal data collected by the biosignal sensor 102. In some embodiments, the processor 108 may be configured to process data collected by the IMU sensor 103 or the air pressure sensor 104.

[0035] In some embodiments, device 100 may include device controller 109. Device controller 109 may be configured to interface with an auxiliary device. For example, device controller 109 may be configured to transmit user commands from device 100 to an auxiliary device. In some embodiments, device controller 109 may be configured to transmit digital content to an auxiliary device.

[0036] In some embodiments, the device 100 may include an audio feedback module 110. The audio feedback module 110 may be configured to output digital audio content to the user. In some embodiments, the module 110 may be configured to output digital content based on the bio-signals of the user collected by the bio-signal sensor 102. In some embodiments, the module 110 may be configured to output digital content based on data received from an external source such as an auxiliary device (e.g., a mobile phone). In some embodiments, the module 110 may be configured to output digital content based on data received from a processor such as a processing unit 111.

[0037] like Figure 1As shown, the processing unit 111 can be configured to receive data from the head-based wearable device 101. The processing unit 111 can also be configured to transmit data to an external server, such as an external server exemplified by the cloud I / O server 121. In some embodiments, the processing unit 111 may include an air pressure sensor 119. In some embodiments, the processing unit 111 can collect air pressure data from the user using the air pressure sensor 119. The processing unit 111 may include a smartphone or computing device. The processing unit 111 may include a laptop, a tablet computer, a mobile computing unit, or a cloud-based computing unit. The processing unit 111 may include a charging box for wireless earbuds. The processing unit 111 may include a GPS transceiver 112, a communication I / O module 113, and an internal processor 116. In some embodiments, the processing unit 111 may include an air pressure sensor for the air pressure sensor 119. In some embodiments, the processing unit may be a cloud-based server.

[0038] The GPS transceiver 112 may be configured to collect GPS location data. In some embodiments, the GPS transceiver 112 may be configured to collect GPS data from physical objects in its vicinity. In some embodiments, the GPS transceiver 112 may be configured to receive GPS data from a user of the head-based wearable device 101.

[0039] The communication I / O module 113 may be configured to transmit data to or receive data from the head-based wearable device 101. In some embodiments, the I / O module 113 may be configured to send or receive data from the cloud I / O server 121. In some embodiments, the I / O module 113 may be, for example, a Bluetooth or similar transceiver suitable for connecting to the head-based wearable device 101. In some embodiments, the I / O module 113 may be a Wi-Fi or cellular data transceiver with an Internet connection. In some embodiments, the I / O module 113 is capable of receiving remote data from the cloud I / O server 121. In some embodiments, the processing unit 111 may include one or more air pressure sensors 119. In some embodiments, the air pressure sensor 119 may include multiple air pressure sensors, wherein one air pressure sensor may be selected as a default. In some embodiments, multiple air pressure sensors may be used to calculate a real-time average.

[0040] The processor 116 may be configured to perform computational analysis based on data received from the device 101. In some embodiments, the processor 116 may be configured to perform computational analysis based on data received from the cloud VO server 121.

[0041] In some embodiments, processing unit 111 may also include a server interface module 114 and / or an object interface module 115. In some embodiments, processing unit 111 may also include a sight vector object (SVO) execution value receiver 117. In some embodiments, unit 111 may also include a device command output module 118.

[0042] The server interface module 114 may be configured to receive data from or transmit data to a cloud I / O server, as shown at 121. The object interface module may be configured to transmit user commands to objects near the user-worn device 100. The object interface module 115 may also be configured to receive data from objects near the user-worn device 101.

[0043] The Sight Vector Object (SVO) execution value receiver 117 may be configured to receive an execution value from the cloud I / O server 121. In some embodiments, the receiver 117 may also receive an execution command from the device 101.

[0044] In some embodiments, the device command output module 118 may send a command to an external device (eg, a cell phone, a television, or a smart thermostat) based on the execution value received via the receiver 117 .

[0045] In some embodiments, processing unit 111 may include a cloud-hosted computing unit in a cloud environment. The cloud environment may include a cloud computing platform, consistent with the disclosed embodiments. Suitable cloud computing platform examples include, but are not limited to, Microsoft AZURE, Amazon Web Services, Google Cloud Platform, IBM CLOUD, and similar systems.

[0046] like Figure 1 As shown, the cloud input / output (I / O) server 121 can be configured to identify a sight vector object (SVO) definition associated with a physical object near the user's wearable device 101. The server 121 can also be configured to match the user data parameters with the SVO definition associated with the physical object near the user. The server 121 can also be configured to determine an execution value based on the SVO definition matching a set of user parameters. The server 121 can also be configured to access the sub-command gateway interface module 125.

[0047] The cloud I / O server 121 may include, but is not limited to, an input server 121a or an output server 121b. In some embodiments, the cloud input / output (I / O) server 121 may include a single server used as the input server 121a. In some embodiments, the server 121 may include an output server 121b. In some embodiments, the input server 121a may include a user / SVO data matching module 122 or a user contact determination module 123. In some embodiments, the output server 121b may include an SVO execution value determination module 124 or a sub-command gateway interface module 125.

[0048] The sight vector object (SVO) database 130 may include a data structure 131 for storing SVO definitions and parameters. The data structure 131 may include a linear data structure, including but not limited to one or more of a table, an array, and a linked list. The data structure may include a non-linear data structure, including but not limited to a graph data structure and / or a tree data structure. The SVO database 130 may include but is not limited to a MySQL database or a NoSQL database, such as Cassandra. In some embodiments, the SVO database 130 may be located in a local processor, such as but not limited to the processing unit 111. In some embodiments, the SVO database 130 may be located in a cloud-based server, such as but not limited to the cloud I / O server 121. In some embodiments, the SVO database 130 may include a local database. In some embodiments, the SVO database 130 may include a distributed database.

[0049] As known to those skilled in the art, Figure 1 The particular arrangement of components shown is not limiting. Consistent with the disclosed embodiments, the system 100 may include additional components or fewer components. For example, the cloud I / O server 121 may include multiple servers, such as an input server 121a and an output server 121b. In some embodiments, the processing unit 111 may be located in a head-based wearable charging box. The processing unit 111 may also be located in an auxiliary computing device, such as a smartphone or computing device, including but not limited to a laptop computer, a tablet computer, a mobile computing unit, or a cloud-based computing unit.

[0050] Figure 2An exemplary method 200 for using a head-based wearable device to interact with a real-world physical object based on a line of sight vector object matrix is ​​described, consistent with the disclosed embodiments. In step 201, user data parameters can be received via a head-based wearable device, such as a head-based wearable device 101. The received user data parameters can include any data collected by the head-based wearable device. The user data parameters received in step 201 can include, for example, data values, which include one or more of pitch, roll, and yaw values. Based on the position and movement of the head-based wearable device, the received user data parameters can also represent the user's position and movement, such as GPS values, altitude values, and / or speed values. The received user data parameters can also include inertial measurement unit (IMU) data or gyroscope data. The user data parameters can also include a set of bio-signals. The set of bio-signals can also include air pressure data, electromyography (EMG) data, electroencephalography (EEG) data, gesture data, and / or facial structure data of the user.

[0051] In step 202, physical object data parameters are received via a head-based wearable device, such as head-based wearable device 101. In some embodiments, the physical object data parameters may include GPS coordinates associated with each object. In some embodiments, the physical object data parameters may include GPS longitude minimum and maximum values, and / or GPS latitude minimum and maximum values. In some embodiments, the physical object data parameters may include altitude values, such as but not limited to minimum altitude values ​​and maximum altitude values. In some embodiments, the physical object data parameters may include speed data. The physical object data parameters may include speed data derived from GPS data. The physical object data parameters may include speed data, the speed data including a minimum speed value and a maximum speed value.

[0052] In step 206, the user's head-based wearable sensor real-time data is transmitted to the processing unit 111. The processing unit 111 can receive the real-time sensor data from the device 101 via any kind of data connection. In some embodiments, the processing unit 111 can receive the IMU data from the IMU sensor 103 within the device 101. In some embodiments, the processing unit 111 can receive the user's real-time biosignal data from the biosignal sensor 102 within the device 100. In some embodiments, the processing unit 111 can receive the user's real-time air pressure data from the air pressure sensor 104 within the device 101. In some embodiments, the processing unit 111 can also receive the user's real-time air pressure data from the air pressure sensor 119. In some embodiments, the processing unit 111 can also receive the user's real-time GPS coordinate data from the GPS transceiver 112. In some embodiments, the device 101 can send real-time physical object parameter data to the processing unit 111. The processing unit 111 receives the real-time physical object parameter data from the device 101 via any kind of data connection.

[0053] In step 208, based on the sensor data and the physical object data parameters from the device 101, it is determined whether the user is near the physical object. The determination may be performed, for example, by the processing unit 111. In step 210, the processing unit 111 of the system 100 may identify a sight vector object matrix (SVOM) associated with the user of the head-based wearable device 101. The processing unit 111 may identify the SVOM based on the physical object data parameters and the SVO definition stored in the data structure 131 of the SVO database server 130. In some embodiments, the processing unit 111 may receive a cloud-hosted SVO definition. In some embodiments, if the SVO definition is remotely hosted, the device 101 may transmit real-time GPS data of the device 101 of the user 101 to the processing unit 111. The processing unit 111 may use an internal processor 116 to process the retrieved SVO definition and physical object data parameters. The sight vector object matrix (SVOM) may include a discrete set or population of SVO definitions, each of which maps to a specific real-world physical object. In some embodiments, users need to qualify for SVOM by choosing to engage with or "opt-in" to the SVOM's services. In some embodiments, SVOMs may be public, private, or a mix of public and private, based on user preferences. In some embodiments, eligibility for SVOMs may involve monetized services. In some embodiments, eligibility for SVOMs may involve military clearance levels (e.g., civilians to military combatants may include different clearance levels associated with different SVOMs). For example, Figure 5Different SVOMs associated with different types of user access are shown, where each SVOM is associated with a different set of SVO definitions associated with a particular physical object. An SVO definition within an SVOM may include a set of data parameters associated with a physical object mapped to that particular SVO definition. Figure 4 A non-limiting example of this set of data parameters is shown in .

[0054] In step 212, the processing unit 111 may determine the user contact state with a specific object based on the SVO associated with the object, the SVOM associated with the user, the user data parameters, and the object data parameters. The processing unit 111 may send the user parameter data from the device 101 data. In some embodiments, the processing unit 111 may transmit the physical object parameter data to the cloud input server 121a. In some embodiments, the processing unit 111 may transmit the SVOM data to the cloud input server 121a. In some embodiments, the user / SVO matching module 122 may determine a set of qualified SVO definitions based on the user data parameters and the physical object data parameters. In some embodiments, the user / SVO matching module 122 may determine a set of qualified SVO definitions based on whether the user's real-time GPS coordinates meet the criteria. In some embodiments, the user's real-time GPS coordinates may be obtained from the GPS transceiver 112 within the scope of influence (AOE) parameters of the qualified SVO definitions defined in the data structure 131 in the SVO database 130. In some embodiments, the user / SVO matching module 122 may determine a set of qualified SVO definitions based on the matching SVO definitions in the SVOM associated with the user's access rights level and type.

[0055] In step 212, the user contact determination module 123 of the input server 121a may determine the contact state between the user and the SVO within the SVOM associated with the user. In some embodiments, the user contact determination module 123 may allow or deny the user to interact with the SVO based on whether the user has access to the SVOM to which the SVO is assigned. For example, if the user does not subscribe to the SVOM and / or is denied access to the SVOM, the user contact determination module 123 may not consider the SVO qualified and may not process the SVO-user data parameter contact. In some embodiments, the user contact determination module 123 may determine the contact state based on a set of allowable user contact data associated with each SVO definition. In some embodiments, a set of allowable user contact data includes a set of duration timestamps, minimum user gaze trigger values, and influence range trigger values ​​associated with the SVO definition. In some embodiments, the user contact determination module 123 may dynamically enable the SVO within the user's SVOM. In some embodiments, the module 123 may enable the SVO by determining whether the user's real-time GPS coordinates are within the predefined influence range (AOE) parameters of the SVO. In some embodiments, the module 123 may dynamically disable the SVO within the user's SVOM. In some embodiments, module 123 may disable the SVO by determining whether the user's real-time GPS coordinates are within a predefined range of influence (AOE) parameter of the SVO. In some embodiments, module 123 may determine an active contact state. In some embodiments, module 123 may determine contact based on when the SVO's latitude minimum and latitude maximum are equal and when the SVO's longitude minimum and longitude maximum are equal. In some embodiments, module 123 may further determine contact when both the SVO's latitude and longitude data are equal to the user's latitude and longitude. In some embodiments, module 123 may determine contact based on the SVO and the user's velocity being equal to zero. In some embodiments, module 123 may determine an active contact state when the SVO's latitude minimum and latitude maximum are not equal and / or the SVO's longitude minimum and longitude maximum are not equal (i.e., the SVO is greater than 1 GPS cubic unit) and the SVO's velocity is equal to zero. In some embodiments, module 123 may determine an active contact state when the SVO's latitude minimum and latitude maximum are equal, the SVO's longitude minimum and longitude maximum are equal, and both are equal to the user's latitude and longitude, and both continue to match the user's latitude and longitude even if the user is moving. In some embodiments, module 123 may determine an active contact state when the SVO's latitude minimum and latitude maximum are dynamic, the SVO's longitude minimum and longitude maximum are dynamic, and the user's latitude and longitude are not equal to the SVO's latitude and longitude regardless of whether the user is moving or stationary.

[0056] After step 212, method 200 may proceed to step 214. In step 214, the SVO execution value determination module 124 of the output server 121b may determine an SVO execution value for each qualified SVO definition mapped to a real-world physical object for the user based on the qualified user-object contact state. The SVO execution value may include a digital integer value, a binary value, or a Boolean value. In some embodiments, if there is an active user contact state between the user and a particular SVO, the execution value is set to an integer greater than 0. In some embodiments, the execution value may correspond to an option that will enable the user to automatically navigate the user to a 3D line type style defined by a Z-axis value. For example, U.S. Patent No. 9,405,366 discloses an example of such a style.

[0057] After step 214, method 200 may proceed to step 216. In step 216, output server 121b may send an execution value based on the user contact state and associated user data and SVO data to processing unit 111. In some embodiments, SVO execution value receiver module 117 of processing unit 111 may receive the data. In some embodiments. Device command output module 118 may process the received data to output commands to auxiliary devices predefined by the SVO definition. In some embodiments, auxiliary devices may include, but are not limited to, IoT devices such as mobile phones, personal computers, tablet computers, televisions, or "smart" thermostats.

[0058] After step 214, method 200 may also proceed to step 218. In step 218, output server 121b may transmit an execution value based on the user contact state and associated user data and SVO data to processing unit 111. SVO execution value receiver module 117 of processing unit 111 may receive the data, and device command output module 118 may process the received data to generate audio feedback data. Output module 118 may send audio feedback data to audio feedback module 110 of device 101. In some embodiments, audio feedback data may include pre-recorded audio narration providing information associated with the user contact object. For example, if the user contact object is a statue of Martin Luther King, the associated audio feedback data may include a pre-recorded audio recording narrating relevant historical information associated with the statue. In some embodiments, audio feedback data may be freely accessible. In some embodiments, audio feedback data may be sponsored by a third party, or if SVO is a commercial entity, it may be sponsored by the SVO object.

[0059] After step 214, method 200 may also proceed to step 220. In step 220, sub-command gateway interface module 125 of output server 121b enables sub-command gateway. In some embodiments, sub-commands may be in the form of up, down, left, right, scroll left, or scroll right, each direction corresponding to one or more predefined execution values.

[0060] Figure 3 An exemplary device as part of a system for interacting with a physical object using a head-based wearable device is described, consistent with the disclosed embodiments. As will be appreciated by those skilled in the art, Figure 1 The specific arrangement of components described in is not intended to be limiting. For example, consistent with the disclosed embodiments, system 100 may include additional components or fewer components. For example, processing unit 111 may include multiple devices, such as mobile phone 1, laptop computer 2, tablet computer 4, or cloud-based computing unit 8.

[0061] Figure 4 An exemplary set of sight vector object (SVO) parameters within a sight vector object matrix (SVOM) as part of a system for interacting with physical objects using a head-based wearable device, consistent with the disclosed embodiments, are described. Figure 4A non-exhaustive list of parameters in may be part of an SVO definition within an SVOM, where each SVO definition is mapped to a corresponding physical object. In some embodiments, an SVO definition corresponding to a real-world physical object (e.g., a television, a game console, or a thermostat) may include an SVOM ID, which includes an identification value of the specific SVOM that includes the SVO, and a gaze vector ID, which includes an identification value of the SVO. In some embodiments, the SVO definition may include a pair of minimum-maximum values ​​for each of the heading, roll, pitch, and yaw (RPY) values ​​of the user collected from the biosignal sensor 102, the IMU 103, and / or the barometric pressure sensor 104 of the device 101, which may collectively form data describing the user's head position and spatial orientation. In some embodiments, the SVO definition may include a pair of minimum-maximum values ​​of the GPS longitude and latitude values ​​collected by the head-based wearable device 101, which may collectively form data describing the user's GPS position. In some embodiments, the SVO definition may include a pair of minimum-maximum values ​​of the altitude or velocity of both the user and the SVO, which values ​​may be used collectively by the processing unit 111 to determine the user's contact state with the SVO. In some embodiments, the SVO definition may include start and end timestamps for allowed contact, which together may define the duration of time that a user is allowed to contact the SVO via the processing unit 111. In some embodiments, the SVO definition may include a minimum gaze trigger time, which may allow the biosignal sensor 102 of the device 101 to measure the number of seconds required for the user to maintain direct gaze on a particular object to qualify as an active contact state, and transmit the user contact state to the processing unit 111 to elicit additional commands. In some embodiments, the SVO definition may include an area of ​​influence (AOE) value, which the user contact determination module 123 of the cloud-based server 121 may utilize to determine the boundaries of the area where the user contact with the SVO is active. In some embodiments, the SVO definition may include an execution value for a sub-command gateway, which enables a user to control an object using the sub-command gateway. In some embodiments, the sub-command gateway may include six sub-commands (e.g., up, down, left, right, scroll left, scroll right).

[0062] Figure 5 An exemplary line of sight vector object matrix is ​​described, which includes various types of access levels and associated line of sight vector object definitions within each matrix, consistent with the disclosed embodiments. For example, SVOM 1 502 describes a publicly accessible SVOM that contains a specific set of SVO definitions ranging from SVO 1 To SVO n SVOM 2504 depicts a private SVOM associated with User 1, which includes a set of different SVO definitions based on the user's access rights; and the SVOM 3 506 describes a military-only SVOM, which may contain a restricted set of SVO definitions that are not accessible to non-military personnel.

[0063] The disclosed embodiments may be implemented in systems, methods, and / or computer program products, which may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0064] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as being a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0065] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0066] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, programming languages ​​include object-oriented programming languages, such as Smalltalk, C++, etc., and conventional program programming languages, such as "C" programming language or similar programming languages. The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In certain embodiments, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions, to personalize the electronic circuit, thereby performing aspects of the present invention.

[0067] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams and the combination of blocks in the flowchart illustrations and / or block diagrams can be implemented by computer-readable program instructions.

[0068] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, thereby generating a method for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram through the instructions executed by the processor of the computer or other programmable data processing device. Therefore, the computer-readable program instructions may also be stored in a computer-readable storage medium, which may instruct the computer, programmable data processing device, and / or other equipment to operate in a specific manner, so that the computer-readable storage medium storing the instructions includes a product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0069] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device to cause a series of operating steps to be performed on the computer, other programmable device, or other device, thereby producing a computer-implemented process, so that the instructions executed on the computer, other programmable device, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0070] The flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a software program, a code segment or a partial code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may not appear in the order marked in the figure. For example, the two boxes shown in succession can actually be executed substantially simultaneously, or these boxes can sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each box in the block diagram and / or flow chart illustration and the combination of boxes in the block diagram and / or flow chart illustration can be implemented by a combination of a dedicated hardware-based system or dedicated hardware and computer instructions that performs a specified function or action.

[0071] The description of various embodiments of the present invention is for illustrative purposes only, but is not intended to be exhaustive or to limit the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements to technologies found on the market, or to enable other persons of ordinary skill in the art to understand the disclosed embodiments.

[0072] It is anticipated that during the patent life of this application, many associated virtualization platforms, virtualization platform environments, trusted cloud platform resources, cloud-based assets, protocols, communication networks, security tokens, and authentication certificates will be developed, and the scope of these terms is intended a priori to include all such new technologies.

[0073] It should be understood that certain features of the invention described in the context of separate embodiments for the sake of clarity may also be provided in combination in separate embodiments. Conversely, various features of the invention described in the context of separate embodiments for the sake of brevity may also be provided separately, or in any suitable sub-combination, or as appropriate in any other described embodiment of the invention. Certain features described in the context of various embodiments should not be considered essential features of those embodiments unless the embodiment is non-functional without those elements.

[0074] Although the present invention has been described in conjunction with its specific embodiments, it is evident that many alternatives, modifications and variations are apparent to those skilled in the art. Therefore, it is intended to cover all alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

Claims

1. A method of interacting with a physical object, comprising: receiving data parameters associated with the user via the head-based wearable device; receiving, via the head-based wearable device, data parameters associated with the object; determining that the user is near the object; as well as transmitting the user data parameter and the object data parameter to a processor, wherein the processor is configured to: identifying at least one sight vector object definition using the object based on the object data parameters; identifying, using the user, at least one line of sight vector object matrix; determining a user contact status with the object; Setting an execution value based on the user contact state; and The execution value is transmitted to an output server.

2. The method according to claim 1, wherein: The user data parameters include one or more of a set of pitch, roll, yaw values, a set of GPS values, an altitude value, a speed value, an inertial measurement unit (IMU) data, and a set of bio-signals.

3. The method according to claim 2, wherein: The set of bio-signals further includes air pressure data, electromyography data, or facial structure data of the user.

4. The method according to claim 1, wherein: The sight vector object definition includes an object ID, a set of position data, and a set of permissible user contact data.

5. The method according to claim 4, wherein: The set of position data defined by the line of sight vector includes longitude and latitude GPS data and speed data derived from the GPS data.

6. The method according to claim 4, wherein: The set of permissible user contact data includes a set of duration timestamps, a minimum user gaze trigger value, and a range of influence trigger value associated with the gaze vector object definition.

7. The method according to claim 1, wherein: The line of sight vector object matrix includes the line of sight vector object definitions using at least the object identification.

8. The method according to claim 1, wherein: The user contact state is determined based on the user data parameters, the gaze vector object definition using the object identification, and the gaze vector object matrix using the user identification.

9. The method according to claim 1, wherein: The output server is configured to be a gateway for the user to enable sub-commands.

10. The system according to claim 9, wherein: The processor unit is further configured to send a command input to a device specified by the gaze vector object definition, or to output digital content to the user via the head-based wearable device.

11. A system for interacting with a physical object, comprising: Head-based wearable devices, memory for storing instructions, and A first processor configured to execute the instructions to: receiving data parameters associated with a user via the head-based wearable device; receiving, via the head-based wearable device, data parameters associated with the object; determining that the user is near the object; transmitting the user data parameters and the object data parameters to the second processor, Wherein, the second processor is configured as follows: identifying at least one sight vector object definition using the object based on the object data parameters; identifying at least one sight vector object matrix using the user; determining a user contact status with the object; Setting an execution value based on the user contact state; and The execution value is transmitted to an output server.

12. The system according to claim 11, wherein: The user data parameters include one or more of a set of pitch, roll, yaw values, a set of GPS values, an altitude value, a speed value, an inertial measurement unit (IMU) data, and a set of bio-signals.

13. The system according to claim 12, wherein: The set of bio-signals further includes air pressure data, electromyography data, or facial structure data of the user.

14. The system according to claim 11, wherein: The sight vector object definition includes an object ID, a set of position data, and a set of permissible user contact data.

15. The system of claim 14, wherein: The set of position data defined by the line of sight vector includes longitude and latitude GPS data and speed data derived from the GPS data.

16. The system of claim 14, wherein: The set of permissible user contact data includes a set of duration timestamps, a minimum user gaze trigger value, and a range of influence trigger value associated with the gaze vector object definition.

17. The system of claim 11, wherein: The line of sight vector object matrix includes the line of sight vector object definitions using at least the object identification.

18. The system of claim 11, wherein: The user contact state is determined based on the user data parameters, a gaze vector object definition using the object identification, and the gaze vector object matrix using the user identification.

19. The system of claim 11, wherein: The output server is configured to be a gateway for the user to enable sub-commands.

20. The system of claim 19, wherein: The second processor is further configured to send a command input to a device specified by the gaze vector object definition, or to output digital content to the user via the head-based wearable device.

Images