Computer software module arrangement, optical see-through device, and method for providing an improved extended reality interface

Abstract

An optical see-through OST arrangement (100) includes a gaze tracking sensor (103), a directional radar sensor (104) configured to guide a beam of radio waves, a memory (102) configured to store a map (M), and a controller (101), wherein the OST device is arranged to be worn on the user's head, wherein the controller (101) is configured to: receive gaze tracking information from the gaze tracking sensor, determine the gaze direction, and in response thereto, adapt the radar sensor based on the gaze tracking information to form a radar beam in the gaze direction.

Description

Technical Field

[0001] This invention relates to an arrangement, including a computer software module, an optical optics OST device, and a method for providing an improved optical optics interface, and more specifically, to an arrangement including a computer software module, an arrangement including circuitry, and an apparatus and method for providing an improved optical optics interface adapted to the user's line of sight. Background Technology

[0002] Wireless communication devices are becoming increasingly sophisticated. Radio access protocol functionality has evolved over time to enable wireless communication using large bandwidths, and wireless devices typically support several different radio frequency bands—sometimes even operating some of these bands simultaneously. As an emerging feature, wireless communication devices can also leverage their advanced capabilities in radio signal transmission, reception, and signal processing to support radar sensing functions. Patent application WO2022008063A1 discloses a communication device, such as a wireless device, equipped with radar sensing capabilities.

[0003] OST devices (such as OST glasses) are becoming increasingly common and radar is also used in these devices, and they are being used over longer periods of time and in different locations. Summary of the Invention

[0004] Following insightful and creative reasoning, the inventors realized that OST glasses would likely incorporate radar sensing capabilities and would require simultaneous localization and mapping (SLAM) during normal operation, such as creating a six-degree-of-freedom (x, y, z, pitch, roll, yaw) position estimate of the OST device relative to visual landmarks in 3D space (i.e., a map). It is well known that various sensors (e.g., single-phase or stereo cameras, lidar, RADAR, etc.) can be used to perform SLAM. As the inventors have recognized, the use of radar can be selected based on where the user is looking, in order to provide better navigation by being able to adapt the radar settings.

[0005] As the inventors are aware, techniques for performing eye tracking are known and have been significantly developed over the past few decades. Eye tracking has proven to be an effective means of determining the direction of a user's gaze, indicating what they are looking at, and subsequently using this information to control devices (e.g., laptops) within a short distance (30cm to 1 meter) using products from companies such as Tobii (www.tobii.com). Wireless devices can be coupled to companion devices with eye-tracking capabilities (e.g., optical see-through OST headphones, glasses, etc.), or in other types of devices, eye-tracking capabilities can be included within the wireless device itself.

[0006] Therefore, the inventors proposed using eye tracking to provide improved adaptation to radar in OST devices.

[0007] The purpose of this paper is to overcome, or at least reduce or mitigate, the problems discussed herein. The inventors have realized that by incorporating eye tracking, radar device settings can be adapted to provide better sensing in that direction, thereby improving map generation or navigation in that direction, as the radar sensor will be enabled to provide more detailed sensing in that direction. Specifically, the inventors have realized that if a user is looking at an object, the radar settings can be adapted accordingly to obtain detailed sensing of that object.

[0008] Therefore, the inventors proposed using posture and / or line of sight to select which configuration to use for radar.

[0009] According to one aspect of the present invention, an optical vision OST device is provided, the OST device including a gaze tracking sensor, a directional radar sensor (104) configured to guide a beam of radio waves, a memory configured to store a map (M), and a controller, wherein the OST device is arranged to be worn on the head of a user, wherein the controller is configured to: receive gaze tracking information from the gaze tracking sensor, determine the gaze direction, and in response thereto, adapt the radar sensor based on the gaze tracking information to form a radar beam in the gaze direction.

[0010] Other embodiments are discussed below, and these other embodiments are also consistent with the appended claims.

[0011] According to another aspect, a method is provided for use in the optical see-through OST device herein, wherein the method includes: receiving gaze tracking information from a gaze tracking sensor, determining a gaze direction, and in response thereto, adapting a radar sensor based on the gaze tracking information to form a radar beam in the gaze direction.

[0012] According to another aspect, a computer-readable medium carrying computer instructions is provided, which, when loaded into and executed by the controller of an OST device, enable the OST device to perform the method according to this document.

[0013] According to another aspect, an optical perspective OST device including a software code module arrangement is provided herein, wherein the software code module arrangement includes software components for receiving gaze tracking information from a gaze tracking sensor, software components for determining the gaze direction, and software components for adapting a radar sensor based on the gaze tracking information in response thereto to form a radar beam in the gaze direction.

[0014] In the context of this teaching, software code modules can be replaced or supplemented by other software modules. Alternatively, in the context of this teaching, software code modules can be replaced or supplemented by circuitry used to perform the corresponding functions.

[0015] Some of the advantages and benefits taught in this paper include: due to limited resources (device power / battery capacity, spectrum, etc.), scanning is performed on the most relevant areas, and additionally, emphasis is placed on scanning the most important areas first and / or with optimal sensing accuracy. Some specific advantages include: the system design and / or operation can automatically adapt to user needs without requiring manual changes to the radar configuration. This faster adaptation allows for the full potential of the radar system. Power savings are achieved by adapting to short-range or lower refresh rate requirements. This provides reduced congestion in the radar's operating band, thus reducing overall interference for users of such systems. Furthermore, the benefits of TX beamforming can be realized without scanning in directions not being viewed by the user. This includes better resolution and distance in the line-of-sight direction and a significantly improved refresh rate.

[0016] Advantages also include: SAR imaging can be used for significantly improved RADAR imaging without the need for manual configuration of user equipment, and performance is enhanced due to corrected visual cues for more appropriate user movement.

[0017] Other embodiments and advantages of the invention will be set forth in the detailed description. It should be noted that the teachings herein find application in see-through devices, such as OST devices (e.g., OST glasses, smartphones, tablet computers, media devices), and even in automotive displays. Attached Figure Description

[0018] Embodiments of the invention will be described below with reference to the accompanying drawings, which illustrate non-limiting examples of how the inventive concept can be put into practice.

[0019] Figure 1A A schematic diagram of a general device to be included in or connected to an OST device according to some embodiments of the present invention is shown.

[0020] Figure 1B A schematic diagram of an OST device according to some embodiments of the present invention is shown.

[0021] Figure 2A and Figure 2B Each of the above illustrations shows a schematic diagram of an OST device according to some embodiments of the teachings herein.

[0022] Figures 3A to 3E Each of the above illustrations shows a schematic diagram of an OST device according to some embodiments of the teachings herein.

[0023] Figures 4A to 4CEach of the above illustrations shows a schematic diagram of an OST device according to some embodiments of the teachings herein.

[0024] Figure 5 A flowchart of a general method according to some embodiments of the present invention is shown.

[0025] Figure 6 A component diagram illustrating the arrangement of software components according to some embodiments of the teachings herein is shown, and

[0026] Figure 7 A schematic diagram is shown of a computer-readable medium carrying computer instructions that, when loaded into and executed by a controller of the arrangement, enable the arrangement to implement some embodiments of the present invention. Detailed Implementation

[0027] Figure 1A A schematic diagram of a general device to be included in or connected to an OST device according to some embodiments of the present invention is shown, both of which will be referred to below as OST device 100. OST device 100 includes a controller 101 and a memory 102. The OST may also include or be connected to a gaze tracking sensor ( Figure 1A Not shown in the image, but... Figure 1B As shown in Figure 1C, labeled 103) and the image rendering device ( Figure 1A Not shown in the image, but... Figure 1B As shown in Figure 1C, labeled 106). The sensor (103) and / or the image rendering device (106) are optional, as one or both of them can be connected to the OST device, and are therefore considered to be included in the OST device by means of connection.

[0028] Controller 101 is configured to control the overall operation of OST device 100. In some embodiments, controller 101 may be a general-purpose controller, where general-purpose means hardware (and / or software) performing various tasks. Those skilled in the art will understand that many alternatives exist for how the controller can be implemented, such as using field-programmable gate arrays, ASICs, GPUs, etc., as alternatives or supplements. For the purposes of this application, all such possibilities and alternatives will be simply referred to as controller 101.

[0029] It should also be noted that in some embodiments, some or all of the controller's processing can be performed remotely, wherein the local controller 101 is configured to provide input data to a remote processing unit (e.g., a cloud server), thereby enabling the remote processing unit to perform processing, and to receive the results of such processing as output from the remote processing unit. For the purposes of this application, this possibility and alternative will be simply referred to as controller 101. Therefore, controller 101 can represent both a local controller and a remote processing unit.

[0030] Memory 102 is configured to store map data, graphics data, user interface (UI) settings, and computer-readable instructions that, when loaded into controller 101, instruct how to control OST device 100. Memory 102 may include multiple storage units or devices, but they will be considered as part of the same overall memory 102. There may be a storage unit for storing graphics data for image rendering devices, a storage unit for storing sensor settings, a storage unit for storing communication interface settings (if any), and so on. Those skilled in the art will understand that there are many possibilities for how to choose where data should be stored. In some embodiments, the OST device may include general-purpose memory 102, and for the purposes of this application, memory 102 further includes any and all such storage units. As those skilled in the art will understand, there are many alternatives for how to implement the memory, such as using non-volatile memory circuitry (e.g., EEPROM memory circuitry) or using volatile memory circuitry (e.g., RAM memory circuitry). For the purposes of this application, all such alternatives will be simply referred to as memory 102.

[0031] In some embodiments, the image presentation device 106 may be a display arrangement including one or more displays configured to primarily present visual data via images. In some such embodiments, the image presentation device 106 may be a touchscreen, thereby enabling user input to be provided to and received by the OST device 100. The visual data is associated with the user interface of the OST device presented by the OST viewing device 100. The OST device is thus arranged to present image data via a (graphical) user interface in a manner controlled by the controller 101.

[0032] In some embodiments, the gaze tracking sensor 103 may be an image sensor (e.g., a camera or image sensor module) arranged to provide an image (or image stream) of the user's environment when the user is using the OST device 100, wherein image processing techniques known in the art can be used to analyze the user's image to determine the user's gaze G, and more specifically, the user's eyes E. Optionally, in some embodiments, the gaze tracking sensor 103 may be other types of photoelectric sensors.

[0033] In some embodiments, the gaze tracking sensor 103 is considered part of the controller 101, wherein gaze (or more precisely, gaze direction) is considered as the user’s posture determined by the controller, as when performing SLAM navigation.

[0034] As those skilled in the art will understand, OST device 100 may include a controller 101 and sensor 103 may include another controller, but for the purposes of this teaching, and to cover all possible changes in determining the exact location where the movement or motion has occurred, they will be regarded as the same controller 101.

[0035] The OST device also includes a radar device 104, which is arranged to sense the surrounding environment by transmitting a beam (labeled as a beam in the figure). In some embodiments, the radar device is based on FMCW technology. FMCW radar (Frequency Modulated Continuous Wave Radar) is a special type of radar sensor that radiates continuous transmission power like a simple continuous wave radar (CW radar). Compared to CW radar, FMCW radar can change its operating frequency during measurement. That is, the transmitted signal is modulated in frequency (or phase). By modulating the signal, the radar can measure the distance to different objects, and the wider the frequency range covered by the modulation, the better the radar can distinguish objects at different distances. For details on how radar sensing functionality is implemented in a wireless device, refer to patent application WO2022008063A1.

[0036] Details of the radar equipment can be found in this design guide for imaging radar from TI (https: / / www.ti.com / lit / ug / tidueq8 / tidueq8.pdf?ts=1696345920543), and its datasheet can be found here (https: / / www.ti.com / lit / ds / symlink / awr2243.pdf?ts=1696272672547). As shown above, these links mention a 13dBm transmitter output power, a 6-phase shifter for TX beamforming, 1.4-degree angular resolution, a maximum range of 150 meters (MIMO), a maximum range of 350 meters (TX beamforming), a maximum speed of 133 km / h, and a speed resolution of 0.53 km / h.

[0037] Radar sensing is a broad technical field. Radar chips can typically be configured to cover several use cases. This configuration requires specifying settings such as transmit power, pulse width and pulse repetition frequency, and operating frequency. For FMCW radar, more commonly used in modern low-cost radar designs (e.g., the automotive industry) and intended for use in OST glasses, scan time, bandwidth, chirp rate, carrier frequency, and parameters used to specify so-called frames—that is, the number of chirps and the time delay between each chirp, and finally, the time delay between frames—can also be adjusted. Additionally, other radar transmit parameters can be adjustable, such as antenna selection and antenna configuration, such as transmit beamwidth. Configuration settings often influence each other, thus requiring balancing to meet the requirements of the use cases. Typically, products using radar chips will include settings for switching between a fixed number of such configurations to best match the possible needs or use cases of the end user.

[0038] In some embodiments, the OST device further includes a forward image sensor 105 configured to provide an image to the controller 101, wherein an object can be determined within the field of view (FOV) of the image sensor 105.

[0039] In some embodiments, the forward image sensor 105 is an alternative to a deduced reckoning sensor, and in some embodiments, the OST device 100 may include other dead reckoning sensors, such as accelerometers, inertial measurement units (IMUs), or other known dead reckoning sensors. In some embodiments, the OST device does not include the forward image sensor 105, but only includes (other) dead reckoning sensors. In some embodiments, the OST device 100 does not include dead reckoning sensors.

[0040] OST device 100 is configured for SLAM, mapping, and / or navigation. SLAM can be performed based on radar equipment, as those skilled in the art will understand. In embodiments where a forward-facing image sensor 105 is arranged in OST device 100, OST device can also be configured for VSLAM (visual SLAM).

[0041] It should be noted that the term "map" will be mentioned several times in the following text, and in all cases, this reference refers to a map created by SLAM or intended to be used by SLAM-enabled devices to locate themselves.

[0042] It should be noted that the teachings herein are applicable to OST devices 100 in many fields of OST (e.g., smartphones, tablets, smartwatches, media devices (e.g., smart TVs), or dedicated OST devices). In some embodiments, OST device 100 may be a smartphone or tablet. In some embodiments, OST device 100 may be a pair of OST goggles 100 (e.g., Figure 1B (As shown).

[0043] Figure 1B A schematic diagram of an OST device 100 according to some embodiments of the present invention is shown, which is an optical vision device (e.g., an OST goggle 100), wherein the OST device is arranged to be worn on a user's head. In some embodiments, the OST device may include more than one image sensor, and according to... Figure 1B In the embodiments (and as those skilled in the art will understand), these are for Figure 1B (Alternative embodiments are also possible), the OST device 100 may be equipped with a forward image sensor 105 for detecting or identifying an object and a backward image sensor 103 for eye tracking.

[0044] In some such embodiments, sensor 103 may be arranged to detect the user's eyes E and detect the user's gaze direction (or simply gaze G), i.e., the direction the user is looking in.

[0045] exist Figure 1B In one embodiment, the image device 106 is a perspective image device, such as a perspective display.

[0046] Figure 1A or Figure 1BThe OST device 100 illustrated may be provided with a communication interface (not explicitly shown, but considered part of the controller 101). The communication interface is configured to communicate with other devices (e.g., other device 100 or a server (not shown)) to receive content, instructions, and / or settings or other data. The communication interface may be wireless or wired. The communication interface may also include several interfaces. In some embodiments, the communication interface may include a USB (Universal Serial Bus) interface. In some embodiments, the communication interface may include an HDMI (High-Definition Multimedia Interface) interface. In other embodiments, the communication interface may include a display port interface. In some embodiments, the communication interface may include an Ethernet interface. In other embodiments, the communication interface may include a MIPI (Mobile Industrial Processor Interface) interface. In other embodiments, the communication interface may include an analog interface, a CAN (Controller Area Network) bus interface, an I2C (Inter-Integrated Circuit) interface, or other interfaces. In some embodiments, the communication interface may include a radio frequency (RF) communication interface. In some such embodiments, the communication interface may include a Bluetooth™ interface, a WiFi™ interface, a ZigBee™ interface, an RFID™ (Radio Frequency Identifier) ​​interface, a WiDi (Wireless Display) interface, a Miracast interface, and / or other RF interfaces typically used for short-range RF communication. In these alternative or supplementary embodiments, the communication interface may include a cellular communication interface, such as a fifth-generation (5G) cellular communication interface, an LTE (Long Term Evolution) interface, a GSM (Global System for Mobile Communications) interface, and / or other interfaces commonly used for cellular communication. In some embodiments, the communication interface may be configured to communicate using the UPnP (Universal Plug and Play) protocol. In other embodiments, the communication interface may be configured to communicate using the DLNA (Digital Living Network Device) protocol. In some embodiments, the communication interface may be configured to communicate via more than one of the example technologies given above. As an example, a wired interface (e.g., MIPI) may be used to establish an interface between the display arrangement, controller, and user interface, and a wireless interface (e.g., WiFi™) may be used to establish communication between the OST device 100 and an external host device (not shown).

[0047] The communication interface can be configured to enable the OST device 100 to communicate with other devices (e.g., other smartphones, internet tablets, computer tablets or other computers, media devices (e.g., televisions), game consoles, video players or projectors (not shown), or glasses detectors) to receive data.

[0048] The following text will also refer to Figure 1A and Figure 1B OST device 100.

[0049] As mentioned above, OST devices are needed to achieve improved navigation, such as by utilizing SLAM. The inventors have realized that by combining eye tracking and radar, the settings of radar device 104 can be adapted to provide better sensing in the direction the user is looking (eye-tracking direction) in order to improve map generation or navigation in that direction, as radar sensor 104 will be enabled to provide more detailed sensing in such directions. Specifically, the inventors have realized that if the user is looking at an object, the radar settings can be adapted accordingly to obtain detailed sensing of that object.

[0050] Therefore, the inventors proposed using posture and / or line of sight to select the configuration of the radar to be used, as referenced herein. Figure 2A , Figure 2B , Figure 3A , Figure 3B , Figure 3C , Figure 4A , Figure 4B and Figure 4C This will be discussed.

[0051] Figure 2A As shown Figure 1A or Figure 1B The OST device 100 includes a gaze-tracking sensor 103 and a radar sensor 104 configured to guide a beam of radio waves, such as a directional radar or an FMCW radar. As shown, the radar device 104 transmits a beam (BEAM) along a first direction. Figure 2A In the example, the first direction is directly forward and is also aligned with the user's gaze G as determined by the gaze tracking sensor 103. As described above, the OST device 100 may also optionally be equipped with a forward image sensor 105, which can be used to determine the object the user is looking at. In embodiments where the OST device is not equipped with a forward image sensor 105, such an object can be determined in other ways, such as by comparing a pose to a map or by actual radar sensing implemented via the radar device 104.

[0052] Therefore, the OST device is configured (via controller 101) to: receive eye-tracking information from the eye-tracking sensor, determine the eye-tracking direction, and, in response, adapt the radar sensor based on the eye-tracking information to form a radar beam in the eye-tracking direction. Figure 2B As shown, the beam has been changed, and in Figure 2B In the example, the direction of the beam has been changed; however, it should be noted that other parameters of the radar beam, not just the direction, can also be changed, or alternatively.

[0053] As is well known, radar sensing is a broad technical field. Radar circuits and systems (specifically, FMCW circuits and systems) can typically be configured to cover several use cases. This configuration requires specifying settings such as transmit power, pulse width and pulse repetition frequency, and operating frequency. For FMCW radar, more commonly used in modern low-cost radar designs (e.g., the automotive industry) and anticipated for XR glasses (as the inventors have already recognized), scan time, bandwidth, chirp rate, carrier frequency, and parameters used to specify so-called frames (i.e., the number of chirps and the time delay between each chirp, and finally, the time delay between frames) can also be adjusted. Additionally, other transmit parameters of the radar can be adjustable, such as antenna selection and antenna configuration, such as transmit beamwidth. Configuration settings often influence each other, thus requiring balancing to meet the requirements of the use cases. Therefore, the OST device 100 according to this document is capable of switching between a fixed number of such configurations to attempt to meet possible needs or use cases. Some examples of use cases may be suitable for slow or stationary targets, while others may be more suitable for estimating the speed of fast-moving targets, detecting nearby or distant targets, creating maps via SLAM, and / or tracking or locating in 3D maps.

[0054] For chips with transmit (TX) beamforming, power can be emitted in a specific direction, and since each direction of interest needs to be illuminated by at least one beam, this will impose many limitations on the overall configuration, especially for narrow beams, as a specific refresh rate is typically expected for radar.

[0055] In some embodiments, the controller 101 is thus configured to adapt the radar operation by selecting a configuration, thereby adapting the beam, wherein the configuration includes one or more settings for transmit power, pulse width, pulse repetition frequency, radio frequency operating frequency, scan time, bandwidth, chirp rate, carrier frequency, radar frames, time delay between frames, transmit antenna and transmit beamwidth, integration time, number of integrated pulses and / or number of transmit antennas used.

[0056] In some embodiments, configuration selection may include adjusting one or more of the following: transmit power, pulse width, pulse repetition frequency, RF operating frequency, scan time, bandwidth and chirp rate, carrier frequency, and parameters for specifying a so-called frame (i.e., the number of chirps and the time delay between each chirp, the time delay between frames, the transmit antenna, and the transmit beamwidth).

[0057] In some embodiments, configuration selection may include selecting to use multiple configurations to perform multiple radar transmissions in different transmission directions relative to a determined line-of-sight direction.

[0058] In some embodiments, configuration selection may include performing SAR processing on radar measurements performed near the line-of-sight direction based on changes in user movement and orientation detected by the IMU and / or radar in different directions.

[0059] In some embodiments, the OST device 100 is further configured to adapt to radar operation by scanning at a first power level in the line-of-sight direction and at a second power level outside the line-of-sight direction.

[0060] In some embodiments, the second power level is 0, i.e., the beam is deactivated.

[0061] In some embodiments, "outside the line of sight" includes directions that exceed an angular difference from the line of sight, wherein the angular difference is 5 degrees, 10 degrees, or 15 degrees.

[0062] Figure 3A Another example is shown where an object (labeled OBJ) is located in front of OST device 100. In some embodiments, OST device 100 is also configured to determine the object (OBJ) in the direction of the line of sight. Therefore, OST device is configured to: determine that an object is present in the direction of the line of sight, and possibly determine aspects or characteristics of the object, such as classifying the object. When a user views the object, it is highly likely that the object is of interest, and as the inventors have already realized, the object should be investigated in order to receive a detailed view of the image. Therefore, in some embodiments, OST device is also configured to: adapt radar sensor 104 based on line-of-sight tracking information to form a radar beam in the direction of the line of sight in order to obtain more detailed sensing of the object. Figure 3B This demonstrates how to adapt a radar beam by orienting it to an object. As mentioned above, adaptation can involve not only changing the direction but also adapting other parameters as a substitute or supplement to the original direction.

[0063] As mentioned above, objects can be identified based on radar sensing.

[0064] As mentioned above, objects can also be determined visually or through other methods, such as based on the user's posture (in conjunction with a map), as a supplement to and alternative to radar sensing.

[0065] In some embodiments, the OST device 100 is further configured to determine objects in the line of sight by receiving image data from an image sensor and using image processing techniques to identify objects.

[0066] In some embodiments, the OST device 100 is further configured to determine objects in the line of sight by: determining the user's posture, determining objects in the map based on the line of sight and the user's posture, and identifying objects in the line of sight as objects in the map.

[0067] In some embodiments, the OST device 100 is further configured to determine objects in the line-of-sight direction by receiving radar data from the radar device 104 and using radar processing technology to determine objects.

[0068] Figure 3A An example is shown where an object OBJ in the user's line of sight has been identified and a distance d from that object has been determined. In some embodiments, the OST device 100 is therefore also configured to: determine the distance to the object in the line of sight direction, and in response thereto, adapt the radar sensor based on the distance to the object in the line of sight direction.

[0069] Figure 3B This illustrates how the beam adapts in a way when the object is located at a first distance D1, and Figure 3C This illustrates how the beam adapts differently in a different way when the object is located at a second distance D2.

[0070] In some embodiments, the OST device 100 is further configured to provide a low-power beam for near objects and a high-power beam for distant objects (D1). <D2)。

[0071] In some embodiments, the OST device 100 is also configured to adapt to integration time, the number of integrated pulses, and / or the number of transmitters used, in addition to the other parameters described herein. For embodiments where the radar device 104 is FMCW, the OST device 100 may also be configured to adapt to bandwidth, which does not inherently affect range; however, it can be a way to shorten integration time. Furthermore, in embodiments where the radar device 104 is FMCW, the OST device 100 may also be configured to adapt to chirp rate, where a higher chirp rate is used for near-field objects. These adaptations also apply to other adaptations discussed herein as possible parameters to be adapted.

[0072] In some embodiments, the OST device 100 is also configured to vary the bandwidth based on distance to change the range resolution. A high resolution will help to see fine details if the user is only viewing very close objects, while a coarser resolution may be sufficient for many use cases when looking at distant objects.

[0073] Figure 3D An example of an object moving is shown, wherein the OST device 100 is also configured to determine the speed of the object in the line of sight direction, and in response thereto, adapt the radar sensor based on the speed of the object in the line of sight direction.

[0074] In some such embodiments, the speed of the object is relative to the speed of the OST device, and in some embodiments, the speed of the object is an absolute speed.

[0075] In some embodiments, velocity may be determined based on the distance covered over time. In some embodiments, velocity may be determined as angular velocity based on the angle covered over time.

[0076] In some embodiments, velocity can be determined using Doppler frequency shift. In some embodiments, a combination of the above methods can be used to determine velocity.

[0077] As described above, in some embodiments, the OST device 100 is configured to determine the velocity of the object relative to the user. This allows for a different adaptation when the user's head moves compared to when only the object moves. In some embodiments, the OST device 100 is configured to separate the angular motion from the object velocity by utilizing dead reckoning sensors to determine the angular motion of the head.

[0078] In some embodiments, the OST device 100 is configured to adapt the beam to provide a wide beam for fast-moving objects and a narrow beam for slow-moving objects. This is in Figure 3D and Figure 3E As shown in the figure, Figure 3D An object moving at a first velocity V1 causes a beam with a first setting, and wherein, Figure 3E An object moving at a second speed V2 results in a beam with a second setting, where V1 is indicated to be lower than V2. Therefore, the determined speed can affect the level of detail sensed, such that fast-moving objects are preferentially kept within the beam, and slow-moving objects are preferentially provided with more detailed sensing, where preference is achieved by selecting parameter settings.

[0079] As described above, in some embodiments, the OST device 100 is configured to determine objects based on the map M. In some embodiments, the OST device is configured to determine which object in the map corresponds to an object identified in the user's line of sight. This can be done by matching the detected object with objects in the map, for example, based on the user's location in the map and the direction and / or distance of the object identified in the user's line of sight.

[0080] Figure 4A This illustrates how objects in map M are matched with objects in the user's line of sight, where an object labeled OBJ A matches an object in the user's line of sight labeled OBJ. In some such embodiments, the matched objects are considered to be objects in the line of sight, such as... Figure 4B As shown. This can be used for navigating relative landmarks (e.g., OBJ A).

[0081] In some embodiments, the OST device 100 is further configured to determine that the level of detail in the line-of-sight direction in the map (e.g., for the detected object OBJ A) exceeds a detail threshold level, and in response to this, to deactivate the radar sensor in the line-of-sight direction to save power.

[0082] In some embodiments, the level of detail threshold is related to the resolution of the data collected for that area of ​​the map. In some embodiments, the level of detail threshold is time-related, wherein newly updated portions of the map do not require thorough scanning as areas that have not been visited for a long time.

[0083] As described above, in some embodiments, the OST device 100 is configured to adapt the map based on the received radar information. In some such embodiments, the OST device 100 is also configured to adapt the radar beam to a high-detail configuration in the line-of-sight direction and supplement the map with data from the dead reckoning sensor 105 using data from the radar sensor. As those skilled in the art will understand, the high-detail configuration will be achieved through high bandwidth and / or narrow beamwidth.

[0084] In some embodiments, and as Figure 4C As shown, the OST device is configured to: determine that there is no object in the map in the line of sight direction, and in response to this, add the object in the line of sight direction to the map.

[0085] Similarly, OST device 100 is configured to determine the presence of an object in the map, but the object is stored at a level of detail below a threshold. In both cases, OST device 100 can also be configured to adapt the beam of a radar sensor to a high-precision beam for receiving detailed radar information about objects in the line-of-sight direction, in order to provide a detailed representation in the map. A high-precision beam can be provided by adapting any, some, or all of the parameters described herein. For example, this can be achieved by adapting the radar transmission to use a wider pulse width, a higher pulse repetition frequency, or increasing the bandwidth of the RF operating band, or by performing sharper TX beamforming in the line-of-sight direction.

[0086] It should be noted that although the above examples mention a transmit beam, the teachings of this paper can also be applied to a receive beam or both. The operational adaptations may be the same or different for either beam.

[0087] Figure 5 A general flowchart of the method taught herein is shown. This method corresponds to the operation of the OST device 100 discussed above, wherein the method includes: receiving 510 gaze tracking information from a gaze tracking sensor, determining 520 a gaze direction, and in response thereto, adapting 530 a radar sensor based on the gaze tracking information to form a radar beam in the gaze direction.

[0088] It should be noted that other functions described herein may include those related to... Figure 5 In some or all of the features discussed, and Figure 5 The method is a general method and also allows other features disclosed above to be implemented as sub-functions of any part of the disclosed method.

[0089] Figure 6 A component diagram of a software component or module arrangement 600 according to some embodiments of the teachings herein is shown. The software component arrangement 600 is adapted for use in the OST device 100 of the teachings herein to provide adaptations to the teachings herein and corresponds to the operation of the OST device 100 described above. The software component arrangement 600 includes: software component 610 for receiving eye-tracking information from an eye-tracking sensor; software component 620 for determining an eye-tracking direction; and software component 630 for adapting a radar sensor based on the eye-tracking information to form a radar beam in the eye-tracking direction in response thereto. In some embodiments, the software component arrangement 600 further includes software component 640 for implementing further functionality discussed in the teachings herein.

[0090] In the context of this teaching, software code modules may be replaced or supplemented by software components. Alternatively, in the context of this teaching, software code modules may be replaced or supplemented by circuits configured to perform corresponding functions.

[0091] Figure 7 A schematic diagram of a computer-readable medium 102 carrying computer instructions 121 is shown, which, when loaded into and executed by the controller of the OST device 100, causes the OST device to implement the teachings herein.

[0092] Computer-readable medium 102 can be tangible, such as a hard drive or flash memory, like a USB memory stick or a cloud server. Alternatively, computer-readable medium 102 can be intangible, such as a signal carrying computer instructions that enable the computer instructions to be downloaded via a network connection (e.g., an internet connection).

[0093] exist Figure 7In the example, computer-readable medium 102 is shown as a computer optical disc 102 carrying computer-readable computer instructions 121 and inserted into a computer optical disc reader 122. The computer optical disc reader 122 may be part of a cloud server 123 or other server, or the computer optical disc reader may be connected to a cloud server 123 or other server. The cloud server 123 may be part of the Internet, or at least connected to the Internet. The cloud server 123 may alternatively be connected via a proprietary or dedicated connection. In one example embodiment, the computer instructions are stored in a remote server 123 and downloaded to the memory 102 of the OST device 100 for execution by the controller 101.

[0094] The computer optical disc reader 122 can also be connected or alternatively connected to (or possibly inserted into) the OST device 100 to transmit computer-readable computer instructions 121 to the controller of the OST device via the memory of the OST viewing device 100.

[0095] Figure 7 The illustration shows both a scenario where OST device 100 receives computer-readable computer instructions 121 via a server connection and a scenario where another OST device 100 receives computer-readable computer instructions 121 via a wired interface. This enables the computer-readable computer instructions 121 to be downloaded to OST viewing device 100, thereby allowing OST device 100 to operate and implement the invention disclosed herein.

Claims

1. An optical vision OST device (100) comprising a gaze-tracking sensor (103), a directional radar sensor (104) configured to guide a beam of radio waves, a memory (102) configured to store a map (M), and a controller (101), wherein, The OST device is arranged to be worn on the user's head, wherein the controller (101) is configured to: Receive eye-tracking information from the eye-tracking sensor Determine the direction of your gaze and respond accordingly. The radar sensor is adapted based on the gaze tracking information to form a radar beam in the gaze direction.

2. The OST device (100) according to claim 1, wherein, The controller (101) is also configured to: Determine the object in the direction of the line of sight, and In response, the parameter settings of the radar sensor are adapted.

3. The OST device (100) according to claim 2, wherein, The controller (101) is also configured to: Determine the distance from the object in the direction of the line of sight, and In response, the parameter settings of the radar sensor are adapted.

4. The OST device (100) according to claim 2 or 3, wherein, The controller (101) is also configured to: Determine the velocity of the object in the line of sight, and In response, the parameter settings of the radar sensor are adapted.

5. The OST device (100) according to any one of claims 2 to 4 further includes an image sensor, wherein, The controller (101) is also configured to: The object in the line-of-sight direction is determined by the following method: Receive image data from the image sensor, and The object is determined using image processing techniques.

6. The OST device (100) according to any one of claims 2 to 5, wherein, The controller (101) is also configured to: The object in the line-of-sight direction is determined by the following method: Determine the user's posture. The objects in the map are determined based on the direction of the gaze and the user's posture, and Objects in the direction of the line of sight are identified as objects in the map.

7. The OST device (100) according to claim 6, wherein, The controller (101) is also configured to determine the distance from an object in the line-of-sight direction.

8. The OST device (100) according to any one of claims 2 to 7, wherein, The controller (101) is also configured to determine objects in the line-of-sight direction by: Radar data is received from radar device 104, and radar processing technology is used to determine the object.

9. The OST device (100) according to any one of claims 2 to 7, wherein, The controller (101) is also configured to: Determine that there are no objects in the map in the stated line of sight direction, and in response to this, Add the objects in the line of sight to the map.

10. The OST device (100) according to claims 7 and 9, wherein, The controller (101) is also configured to determine that no object exists in the map at a distance determined in the line-of-sight direction.

11. The OST device (100) according to any one of claims 2 to 10, wherein, The controller (101) is also configured to adapt the beam of the radar sensor to a high-precision beam for receiving detailed radar information about objects in the line-of-sight direction.

12. The OST device (100) according to any one of the preceding claims, wherein, The controller (101) is also configured to adapt the map based on the received radar information.

13. The OST device (100) according to any one of the preceding claims, wherein, The controller (101) is also configured to: determine that the level of detail in the map in the line-of-sight direction exceeds a detail threshold level, and in response to this, activate the radar sensor in the line-of-sight direction.

14. The OST device (100) according to any one of the preceding claims, wherein, The controller (101) is further configured to adapt the radar beam by scanning at a first power level in the line-of-sight direction and at a second power level outside the line-of-sight direction, wherein the first power level is higher than the second power level.

15. The OST device (100) according to claim 14, wherein, The second power level is 0.

16. The OST device (100) according to claim 14 or 15, wherein, Beyond the line of sight includes directions that exceed an angle difference from the line of sight, wherein the angle difference is 5 degrees, 10 degrees, or 15 degrees.

17. The OST device (100) according to any one of the preceding claims further includes a dead reckoning sensor (105), and wherein, The controller (101) is also configured to adapt the radar beam to a high-detail configuration in the line-of-sight direction and supplement the map with data from the dead reckoning sensor (105) using data from the radar sensor.

18. The OST device (100) according to any one of the preceding claims, wherein, The controller (101) is also configured to adapt the radar beam by selecting a configuration, wherein the configuration includes one or more settings of transmit power, pulse width, pulse repetition frequency, radio frequency operating frequency, scan time, bandwidth, chirp rate, carrier frequency, radar frame, time delay between frames, transmit antenna, and transmit beamwidth.

19. A method for use in an OST device (100), the OST device comprising a gaze-tracking sensor (104), a directional radar sensor (103) configured to guide a beam of radio waves, a memory (102) configured to store a map, and a controller (101), wherein, The OST device is positioned to be worn on the user's head, wherein the method includes: Receive eye-tracking information from the eye-tracking sensor Determine the direction of your gaze and respond accordingly. The radar sensor is adapted based on the gaze tracking information to form a radar beam in the gaze direction.

20. A computer-readable medium (720) carrying computer instructions (721) that, when loaded into and executed by a controller (101) of an OST device (100), enable the OST device (100) to perform the method according to claim 19.

21. A software component arrangement for adapting a radar beam to an OST device (100), the OST device including a line-of-sight tracking sensor (104), a directional radar sensor (103) configured to guide a beam of radio waves, a memory (102) configured to store a map, and a controller (101), wherein, The OST device is positioned to be worn on the user's head, wherein the software components include: Software components for receiving eye-tracking information from the eye-tracking sensor. Software components used to determine the direction of the line of sight, and A software component for responding to this by adapting the radar sensor based on the gaze tracking information to form a radar beam in the gaze direction.

Images