A computer software module arrangement, an optical-see-through device and a method for providing an improved extended reality interface

Abstract

An Optical-See-Through, OST, arrangement (100) comprising a gaze-tracking sensor (103), a directional radar sensor (104) configured for steering a beam of radio waves, a memory (102) configured to store a map (M) and a controller (101), wherein the OST device is arranged to be worn on a user's head, wherein the controller (101) is configured to receive gaze tracking information from the gaze-tracking sensor, determine a direction of gaze, and in response thereto adapt the radar sensor based on the gaze-tracking information to form a radar beam in the direction of the gaze.

Description

[0001] A COMPUTER SOFTWARE MODULE ARRANGEMENT, AN OPTICAL-SEE-THROUGH DEVICE AND A

[0002] METHOD FOR PROVIDING AN IMPROVED EXTENDED REALITY INTERFACE

[0003] TECHNICAL FIELD

[0004] The present invention relates to an arrangement, an arrangement comprising computer software modules, an optical-see-through, OST, device and a method for providing an improved optical- see-through interface, and in particular to an arrangement comprising computer software modules, an arrangement comprising circuits, a device and a method for providing an improved optical-see-through interface which adapts to a direction of gaze of a user.

[0005] BACKGROUND

[0006] Wireless communication devices are becoming more and more advanced. The evolution in radio access protocol functionalities over time has enabled wireless communication using large bandwidths, and the wireless devices typically support several different radio frequency bands - even sometimes operating some of them simultaneously. As an emerging feature, wireless communication devices may also support radar sensing functionalities utilizing its high-end capabilities of radio signal transmission, reception, and signal processing. The patent application published as W02022008063A1 discloses a communication device, such as a wireless device, which is arranged with radar sensing functionalities.

[0007] OST devices, such as OST glasses, are becoming more and more commonplace and also the use of radar in such devices, and also used for longer periods of time and in different locations.

[0008] SUMMARY

[0009] The inventors have realized after insightful and inventive reasoning that OST glasses are likely to be expected to include radar sensing functionality and during normal operation will be required to perform Simultaneous Localization And Mapping (SLAM), e.g. to create a 6 degrees of freedom (x,y,z, pitch, roll, yaw) position estimate of the OST device relative to visual landmarks in a 3D space, i.e., a map. As is known, SLAM can be performed using various sensors such as mono or stereo camera, lidar, RADAR, etc. As the inventors have realized, the use of radar can be adopted based on where the user is looking in order to provide a better navigation through being able to adapt the radar settings.

[0010] As the inventors know, the technology of performing eye tracking is known and has significantly developed the last decades. Eye-tracking has been shown to be an effective means to determine gaze directions indicating what a user is looking at and subsequently use that information to control a device such as a laptop from a short distance (30cm up to a meter) using products from companies such as Tobii (www.tobii.com). Wireless devices may be coupled to companion devices such as optical-see-through ,OST , headsets, glasses etc with eye tracking capability, or in other types of devices eye tracking capability can be included in the wireless device itself.

[0011] The inventors are thus proposing to utilize eye-tracking in order to provide an improved adaptation of a radar in an OST device.

[0012] An object of the teachings herein is to overcome or at least reduce or mitigate the problems discussed herein. The inventors have realized that by combining gaze tracking, the settings for the radar device can be adapted to provide better sensing in such a direction in order to improve the map generation or navigation in that direction as the radar sensor will be enabled to provide a more detailed sensing in such a direction. In particular, the inventors have realized that if the user is looking at an object, the radar settings may be adapted accordingly in order to retrieve a detailed sensing of that object.

[0013] The inventors are thus proposing to utilize the pose and / or gaze to select which configuration to use for the radar.

[0014] According to one aspect there is provided an optical-see-through, OST, device comprising a gaze-tracking sensor, a directional radar sensor (104) configured for steering a beam of radio waves, a memory configured to store a map (M) and a controller, wherein the OST device is arranged to be worn on a user's head, wherein the controller is configured to receive gaze tracking information from the gaze-tracking sensor, determine a direction of gaze, and in response thereto adapt the radar sensor based on the gaze-tracking information to form a radar beam in the direction of the gaze.

[0015] Other embodiments are discussed in the below and are also as per the appended claims.

[0016] According to another aspect there is provided a method for use in an optical-see-through, OST, device as herein, wherein said method comprises receiving gaze tracking information from the gazetracking sensor, determining a direction of gaze, and in response thereto adapting the radar sensor based on the gaze-tracking information to form a radar beam in the direction of the gaze. According to another aspect there is provided a computer-readable medium carrying computer instructions that when loaded into and executed by a controller of an OST device enables the OST device to execute a method according to herein.

[0017] According to another aspect there is provided an optical-see-through, OST, device as herein comprising a software code module arrangement, wherein the software code module arrangement comprises a software component for receiving gaze tracking information from the gaze-tracking sensor, a software component for determining a direction of gaze, and a software component for adapting the radar sensor based on the gaze-tracking information to form a radar beam in the direction of the gaze in response thereto.

[0018] For the context of the teachings herein a software code module may be replaced or supplemented by a software module. For the context of the teachings herein a software code module may alternatively be replaced or supplemented by a circuit for performing a corresponding function.

[0019] Some advantages and benefits of the teachings herein include that since resources (device energy / battery capacity, spectrum, etc) are limited, the most relevant areas are scanned, and in addition the focus on most important areas can be scanned first and / or with best sensing accuracy. Some specific advantages include that the system design and / or operation can automatically be adapted to the user needs without manually changing the radar configuration. That faster changes allow for using the full potential of the radar system. That power can be saved by adapting for close range or lower refresh rate needs. That there is provided decreased congestion on the frequency band the radar is operating on, reducing interference overall for users of such systems. And that the benefits of TX- beamforming can be realized without scanning in directions where the user is not looking. This includes better resolution and distance in the direction of the gaze, significantly improved refresh rate.

[0020] The advantages also include that SAR Imaging can be used for much improved RADAR imaging without manual configuration of the user device and with enhanced performance due to corrective visual cue for more appropriate user movements.

[0021] Further embodiments and advantages of the present invention will be given in the detailed description. It should be noted that the teachings herein find use in see-through devices, such as in OST devices such as OST glasses, smartphones, tablet computers, media devices, and even in vehicular displays.

[0022] BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention will be described in the following, reference being made to the appended drawings which illustrate non-limiting examples of how the inventive concept can be reduced into practice.

[0023] Figure 1A shows a schematic view of a general device to be comprised in or connected to an OST device according to some embodiments of the present invention,

[0024] Figure IB shows a schematic view of an OST device according to some embodiments of the present invention,

[0025] Figures 2A and 2B each shows a schematic view of an OST device according to some embodiments of the teachings herein,

[0026] Figures 3A to 3E each shows a schematic view of an OST device according to some embodiments of the teachings herein,

[0027] Figures 4A to 4C each shows a schematic view of an OST device according to some embodiments of the teachings herein,

[0028] Figure 5 shows a flowchart of a general method according to some embodiments of the present invention,

[0029] Figure 6 shows a component view for a software component arrangement according to some embodiments of the teachings herein, and

[0030] Figure 7 shows a schematic view of a computer-readable medium carrying computer instructions that when loaded into and executed by a controller of an arrangement enables the arrangement to implement some embodiments of the present invention.

[0031] DETAILED DESCRIPTION

[0032] Figure 1A shows a schematic view of a general device to be comprised in or connected to an OST device 100 according to some embodiments of the present invention, Hereafter both will be referred to as the OST device 100. The OST device 100 comprises a controller 101 and a memory 102. The OST may also comprise or be connected to a gaze tracking sensor (not shown in figure 1A, but shown in figures IB and 1C referenced 103) and an image presenting device (not shown in figure 1A, but shown in figures IB and 1C referenced 106). The sensor (103) and / or the image presenting device (106) are optional in that one or both of them may be connected to the OST device, and are thus considered to be comprised in the OST device through the connection. The controller 101 is configured to control the overall operation of the OST device 100. In some embodiments, the controller 101 may be a general-purpose controller, wherein general purpose refers to hardware (and / or software) that perform a variety of tasks. As a skilled person would understand there are many alternatives for how to implement a controller, such as using Field - Programmable Gate Arrays circuits, ASICs, GPUs, etc. in addition to or as an alternative. For the purpose of this application, all such possibilities and alternatives will be referred to simply as the controller 101.

[0033] It should also be noted that in some embodiments, parts of or all of the processing of the controller may be performed remotely, where a local controller 101 is configured to provide input data to a remote processing unit, such as to a cloud server, causing the remote processing unit to perform the processing and receive the results of such processing as output from the remote processing unit. For the purpose of this application, such possibilities and alternatives will also be referred to simply as the controller 101. The controller 101 may thus represent both the local controller and the remote processing unit.

[0034] The memory 102 is configured to store map data, graphics data, User Interface (Ul) settings and computer-readable instructions that when loaded into the controller 101 indicate how the OST device 100 is to be controlled. The memory 102 may comprise several memory units or devices, but they will be perceived as being part of the same overall memory 102. There may be one memory unit for the image presenting device storing graphics data, one memory unit for the sensor for storing settings, one memory for the communications interface (if such is present) for storing settings, and so on. As a skilled person would understand, there are many possibilities of how to select where data should be stored. In one embodiment, the OST device may comprise a general memory 102 in turn comprising any and all such memory units for the purpose of this application. As a skilled person would understand there are many alternatives of how to implement a memory, for example using non-volatile memory circuits, such as EEPROM memory circuits, or using volatile memory circuits, such as RAM memory circuits. For the purpose of this application all such alternatives will be referred to simply as the memory 102.

[0035] The image presenting device 106 may in some embodiments be a display arrangement comprising one or more displays arranged to present visual data, predominantly through images. In some such embodiments, the image presenting device 106 may be a touch screen thereby enabling user input to be provided to and received by the OST device 100. The visual data is related to the user interface of the OST device being presented by the OST viewing device 100. The OST device is thereby arranged to present image data through a (graphical) user interface in a manner controlled by the controller 101.

[0036] The gaze tracking sensor 103 may in some embodiments be an image sensor, such as a camera or image sensor module, arranged to provide an image (or stream of images) of the user environment when the user is utilizing the OST device 100, wherein images of the user may be analyzed using image processing techniques known in the art in order to determine a gaze G of the user, and more specifically the user's eye(s) E. Optionally the gaze tracking sensor 103 may in some embodiments be other types of electro-optical sensors.

[0037] In some embodiments the gaze tracking sensor 103 is considered to be part of the controller 101, wherein the gaze, or rather the direction of gaze is taken to be the pose of the user as determined by the controller, as when performing SLAM navigation.

[0038] As a skilled person would understand, the OST device 100 may comprise one controller 101 and the sensor 103 may comprise another controller, but for the purpose of the teachings herein, they will be considered to be the same controller 101 in order to cover all possible variations of exactly where the determination of movement or motion takes place.

[0039] The OST device also comprises a radar device 104 which is arranged to sense the surroundings by emitting a beam (referenced BEAM in the figures). In some embodiments the radar device is based on FMCW technology. FMCW radar (Frequency-Modulated Continuous Wave radar) is a special type of radar sensor which radiates continuous transmission power like a simple continuous wave radar (CW- Radar). In contrast to a CW radar, the FMCW radar can change its operating frequency during the measurement. That is, the transmission signal is modulated in frequency (or in phase). By modulating the signal it is possible for the radar to measure distance to different objects, and the wider the range of frequencies covered by the modulation the better the radar can resolve objects at different distances from each other. For details on how to implement radar sensing functionalities in a wireless device, reference is made to the patent application published as W02022008063A1.

[0040] Details for a radar device can be found in this Design Guide for an imaging radar from Tl: (https: / / www.ti.com / lit / ug / tidueq8 / tidueq8.pdf?ts=1696345920543) for which a data sheet can be found here (https: / / www.ti.com / lit / ds / symlink / awr2243.pdf?ts=1696272672547). As could be noted, these links mention 13 dBm transmitter output power, 6-bit phase-shifter for TX beamforming , 1.4 degree angular resolution, maximum range 150 m (mimo), maximum range 350m (TX beamforming), max velocity 133 km / h and velocity resolution 0.53 km / h. Radar sensing is a wide technology area. Radar chips can commonly be configured to cover several use-cases. This configuration requires settings to be specified such as transmit power, pulse width and pulse repetition frequency and operating frequency. For FMCW radars, more commonly used in modern low-cost radar designs such as in the automotive industry and expected for OST glasses, one can further adjust sweep time, bandwidth, chirp rate, carrier frequency, as well as parameters specifying the so called frame, i.e., number of chirps and time delay between each chirp, and finally the time delay between frames. Also, other transmit parameters of a radar may be adjustable, e.g. antenna selection and antenna configurations, e.g. transmit beamwidth. The configuration settings typically affect each other so a balance needs to be taken to meet the requirements of the use case. Typically, the product using the radar chip would include a setting to switch between a fixed number of such configurations to try match the end-user's possible needs or use-cases.

[0041] In some embodiments the OST device also comprises a forward-facing image sensor 105 that is configured to provide the controller 101 with images in which an object may be determined within a field-of-view, FOV, of the image sensor 105.

[0042] In some embodiments the forward-facing image sensor 105 is one alternative for a deduced reckoning sensor, and the OST device 100 may, in some embodiments, comprise other deduced reckoning sensors, such as accelerometer, Inertial Measurement Unit (IMU), or other known deduced reckoning sensors. In some embodiments the OST device does not comprise a forward-facing image sensor 105 only (other) deduced reckoning sensors. In some embodiments no deduced reckoning sensors are comprised in the OST device 100.

[0043] The OST device 100 is configured for SLAM, for mapping and / or for navigation. The SLAM may be performed based on the radar device as would be understood by a skilled person. In embodiments where the OST device 100 is arranged with a forward-facing image sensor 105, the OST device may also be configured for VSLAM (Visual SLAM).

[0044] It should be noted that there will be several references to "maps" in the following and in all cases the reference is to be taken to mean maps created by SLAM or to be used for SLAM capable devices to position themselves in.

[0045] It should be noted that the teachings herein find use in OST devices 100 in many areas of OST, such as for example smart phones, tablet computers, smart watches, media devices (such as smart TVs) or specifically in OST devices. In some embodiments, the OST device 100 may be a smart phone or tablet. In some embodiments, the OST device 100 may be a pair of OST goggles 100 (as in figure IB). Figure IB shows a schematic view of an OST device 100 being an optical-see-through device, such as a pair of OST goggles 100 according to some embodiments of the present invention wherein the OST device is arranged to be worn on a user's head. In some embodiments, the OST device may comprise more than one image sensor, and in the embodiments according to figure IB (and as the skilled person would understand, these are also possible for the embodiments of figure IB) the OST device 100 may be arranged with a front facing image sensor 105 for detecting or determining objects as well as a rearward facing image sensor 103 for gaze-tracking.

[0046] In some such embodiments the sensor 103 may be arranged to detect an eye E of the user and to detect a gaze direction or in short a gaze G of the user, i.e. a direction in which the user is looking.

[0047] In the embodiments of figure IB, the image device 106 is a see-through image device, such as a see-through display.

[0048] The OST device 100 as exemplified in either of figures 1A or IB may be arranged with a communication interface (not shown explicitly, but seen as being part of the controller 101). The communication interface is arranged to enable communication with other devices, such as other devices 100 or a server (not shown) for receiving content, instructions and / or settings or other data. The communication interface may be wired and / or wireless. The communication interface may also comprise several interfaces. In some embodiments, the communication interface may comprise a USB (Universal Serial Bus) interface. In some embodiments the communication interface may comprise an HDMI (High Definition Multimedia Interface) interface. In other embodiments, the communication interface may comprise a Display Port interface. In some embodiments the communication interface may comprise an Ethernet interface. In yet other embodiments, the communication interface may comprise a MIPI (Mobile Industry Processor Interface) interface. In other embodiments, the communication interface may comprise an analog interface, a CAN (Controller Area Network) bus interface, an I2C (Inter-Integrated Circuit) interface, or other interfaces. In some embodiments, the communication interface may comprise a radio frequency (RF) communications interface. In some such embodiments the communication interface may comprise a Bluetooth™ interface, a WiFi™ interface, a ZigBee™ interface, an RFID™ (Radio Frequency I Dentifier) interface, Wireless Display (Wi Di) interface, Miracast interface, and / or other RF interface commonly used for short range RF communication. In alternative or supplemental such embodiments, the communication interface may comprise a cellular communications interface such as a fifth generation (5G) cellular communication interface, an LTE (Long Term Evolution) interface, a GSM (Global Systeme Mobile) interface and / or other interface commonly used for cellular communication. In some embodiments, the communication interface may be configured to communicate using the UPnP (Universal Plug n Play) protocol. In other embodiments the communication interface may be configured to communicate using the DLNA (Digital Living Network Appliance) protocol. In some embodiments, the communication interface may be configured to enable communication through more than one of the example technologies given above. As an example, a wired interface, such as MIPI may be used for establishing an interface between the display arrangement, the controller and the user interface, and a wireless interface, for example WiFi™ could be used to enable communication between the OST device 100 and an external host device (not shown).

[0049] The communications interface may be configured to enable the OST device 100 to communicate with other devices, such as other smartphones, Internet tablets, computer tablets or other computers, media devices, such as television sets, gaming consoles, video viewer or projectors (not shown), or eyewear detectors for receiving data.

[0050] In the following, simultaneous reference will be made to the OST devices 100 of figures 1A and IB.

[0051] As mentioned above, there is a need for OST devices to enable for an improved navigation, such as utilizing SLAM. The inventors have realized that by combining gaze tracking and radar, the settings for the radar device 104 can be adapted to provide better sensing in a direction where the user is looking (the gaze direction) in order to improve the map generation or navigation in that direction as the radar sensor 104 will be enabled to provide a more detailed sensing in such a direction. In particular, the inventors have realized that if the user is looking at an object, the radar settings may be adapted accordingly in order to retrieve a detailed sensing of that object.

[0052] The i9nventors are thus proposing to utilize the pose and / or gaze to select which configuration to use for the radar as will be discussed herein with reference to figures 2A, 2B, 3A, 3B, 3C, 4A, 4B and 4C.

[0053] Figure 2A shows an OST device 100, as in figures 1A or IB, comprising a gaze-tracking sensor 103), a radar sensor 104 configured for steering a beam of radio waves, such as a directional radar, for example an FMCW radar. As is illustrated the radar device 104 is emitting a beam (BEAM) in a first direction, which in the example of figure 2A is straight ahead - and also in line with the gaze G of the user as determined by the gaze tracking sensor 103. As discussed in the above, the OST device 100 may optionally be arranged with a forward-facing image sensor 105 as well which may be used to determine an object that the user is looking at. In embodiments where the OST device is not arranged with a forward-facing image sensor 105, such object may be determined in other manner, such as comparing a pose to a map, or through the actual radar sensing achieved through the radar device 104.

[0054] The OST device is thus configured to (through the controller 101) to receive gaze tracking information from the gaze-tracking sensor and to determine a direction of gaze, and in response thereto adapt the radar sensor based on the gaze-tracking information to form a radar beam in the direction of the gaze. As is shown in figure 2B the beam has been changed, and in the example of figure 2B, the direction of the beam has been changed, however, it should be noted that other parameters of the radar beam may also or alternatively be changed, not only the direction.

[0055] As is known, radar sensing is a wide technology area. Radar circuits and systems, especially FMCW circuits and systems, can commonly be configured to cover several use-cases. This configuration requires settings to be specified such as transmit power, pulse width and pulse repetition frequency and operating frequency. For FMCW radars, more commonly used in modern low-cost radar designs such as in the automotive industry and expected for XR glasses (as the inventors have realized), one can further adjust sweep time, bandwidth, chirp rate, carrier frequency, as well as parameters specifying the so- called frame, i.e., number of chirps and time delay between each chirp, and finally the time delay between frames. Also, other transmit parameters of a radar may be adjustable, e.g. antenna selection and antenna configurations, e.g. transmit beamwidth. The configuration settings typically affect each other so a balance needs to be taken to meet the requirements of the use case. The OST device 100 according to herein is thus enabled to switch between a fixed number of such configurations to try match possible needs or use-cases. Some examples of use-cases could be adapted to slow or stationary targets, while others might be better adapted for estimating the velocity of fast-moving targets, detecting targets that are close or far away, creating a map via SLAM and / or tracking or localization in a 3D map.

[0056] For chips with transmitting (TX) beamforming the power can be transmitted in a specific direction, and since each direction of interest needs to be illuminated by at least one beam, this will put a lot of restrictions on the overall configuration, especially for narrow beams, as typically a certain refresh rate is expected of the radar.

[0057] In some embodiments the controller 101 is thus configured to adapt the radar operation, thereby adapting the beam, by selecting a configuration, wherein a configuration includes one or more settings for transmit power, pulse width, pulse repetition frequency, Radio Frequency operating frequency, sweep time, bandwidth, chirp rate, carrier frequency, radar frame, time delay between frames, transmit antennas and transmit beamwidth, the integration time, the number of integrated pulses, and / or the number of transmit antennas utilized.

[0058] In some embodiments the configuration selection may include adjusting one or more of transmit power, pulse width, pulse repetition frequency, RF operating frequency, sweep time, bandwidth and chirp rate, carrier frequency, as well as parameters specifying the so called frame, i.e., number of chirps and time delay between each chirp, time delay between frames, transmit antennas and transmit beamwidth.

[0059] In some embodiments the configuration selection may include selecting to use multiple configurations for multiple radar transmissions in different transmit directions in relation to a determined gaze direction.

[0060] In some embodiments the configuration selection may include performing SAR processing of the radar measurements performed near the gaze direction, based on movements and orientation changes of the user as detected by IMU and / or radar measurements in different directions.

[0061] In some embodiments the OST device 100 is further configured to adapt the operation of the radar by scanning with a first power level in a beam direction of gaze and a second power level outside the direction of gaze, wherein the first power level is higher than the second power level.

[0062] In some embodiments the second power level is 0, i.e. the beam is deactivated.

[0063] In some embodiments outside the direction of gaze includes directions exceeding an angular difference from the direction of gaze, wherein the angular difference is 5, 10 or 15 degrees.

[0064] Figure 3A shows a further example where an object (referenced OBJ) is in front of the OST device 100. In some embodiments, the OST device 100 is further configured to determine an object (OBJ) that is in the direction of gaze. The OST device is thus configured to determine that there is an object in the direction of the gaze and possibly to determine aspects or features of that object, such as classifying the object. As the user is looking at the object, the object is most likely of interest and as the inventors have realized, that object should be investigated in order to receive a detailed view of the image. The OST device is therefore in some embodiments further configured to adapt the radar sensor 104 based on the gaze-tracking information to form the radar beam in the direction of the gaze in order to retrieve a more detailed sensing of the object. Figure 3B shows how the radar beam has been adapted by being directed to the object. As mentioned above, the adaptation may not be to change the direction, but could also be an adaptation of other parameters, alternatively or in addition to adapting the direction. As discussed in the above, the object may be determined based on the radar sensing.

[0065] As discussed in the above, the object may be determined visually or through other determinations, such as based on the pose of the user (in combination with a map) in addition to or as an alternative to the radar sensing.

[0066] In some embodiments the OST device 100 is further configured to determine the object in the direction of gaze by receiving image data from the image sensor and determining the object utilizing image processing techniques.

[0067] In some embodiments the OST device 100 is further configured to determine the object in the direction of gaze by determining a pose of the user, determining an object in the map based on the direction of gaze and the pose of the user and determining the object in the direction of gaze to be the object in the map.

[0068] In some embodiments the OST device 100 is further configured to determine the object in the direction of gaze by receiving radar data from the radar device 104 and determining the object utilizing radar processing techniques.

[0069] Figure 3A shows an example where an object OBJ in the gaze of the user has been determined, and wherein a distance d to the object is determined. In some embodiments the OST device 100 is thus further configured to determine a distance to the object in the direction of gaze and in response thereto adapt the radar sensor based on the distance to the object in the direction of gaze.

[0070] Figure 3B shows how the beam is adapted in one way for an object being at a first distance DI, and figure 3C shows how the beam is adapted differently in another way for an object being at a second distance D2.

[0071] In some embodiments the OST device 100 is further configured to provide a beam of lower power for a close object and provide a beam of higher power for a far-away object (D1<D2)

[0072] In some embodiments the OST device 100 is further configured to adapt the integration time, the number of integrated pulses, and / or the number of transmitters utilized in addition to other parameters mentioned herein. For embodiments where the radar device 104 is FMCW the OST device 100 may also be configured to adapt the bandwidth which does not affect the distance on its own, however, it could be a way to decrease the integration time. Furthermore, in embodiments where the radar device 104 is FMCW the OST device 100 may also be configured to adapt the chirp rate, with a higher chirp rate for close objects. These adaptations also apply to other adaptations discussed herein as possible parameters to adapt. In some embodiments the OST device 100 is further configured to change the bandwidth depending on distance to change the range resolution. If a user is only looking at an object that is very close, it would be useful with a high resolution to see the fine details and when looking far ahead it would probably be sufficient for many use cases to have a coarser resolution.

[0073] Figure 3D shows an example where the object is moving, and wherein the OST device 100 is further configured to determine a speed of the object in the direction of gaze and in response thereto adapt the radar sensor based on the speed of the object in the direction of gaze.

[0074] In some such embodiments the speed of the object is a speed relative the OST device, and in some embodiments the speed of the object is an absolute speed.

[0075] In some embodiments the speed may be determined based on a distance covered over time. In some embodiments the speed may be determined as an angular speed based on an angle covered over time.

[0076] In some embodiments the speed may be determined using the doppler shift. In some embodiments a combination of the above can be used to determine the speed.

[0077] As noted above, in some embodiments the OST device 100 is configured to determine the speed of the object as a speed relative the user. This will allow for a different adaptation to be made when the user's head is moving as compared to when only the object is moving. In some embodiments the OST device 100 is configured to determine an angular movement of the head to separate that from object speed, by utilizing deduced reckoning sensors.

[0078] In some embodiments the OST device 100 is configured to adapt the beam to provide a wide beam for a fast-moving object and to provide a narrow beam for a slow-moving object. This is illustrated in figures 3D and 3E where the object in figure 3D moving at a first speed VI brings about a beam having a first setting and wherein the object in figure 3E moving at a second speed V2 brings about a beam having a second setting, where VI is indicated to be lower than V2. The speed determined can thus affect the level of detail for the sensing so that a fast-moving object is prioritized to stay within a beam and a slow-moving object is prioritized to provide a more detailed sensing, wherein the prioritizing is effected through a selection of settings for parameters.

[0079] As mentioned above, the OST device 100 is in some embodiments configured to determine the object based on a map M. In some embodiments the OST device is configured to determine which object in the map that the object determined in the gaze of the user corresponds to. This can be done by matching the object detected to objects in the map, for example based on the user's location in the map, and a direction and / or a distance to the object determined in the gaze of the user.

[0080] Figure 4A shows how an object in a map M is matched to the object that is in the gaze of the user, wherein the object referenced OBJ A is matched to the object in the gaze of the user referenced OBJ. In some such embodiments the object matched is taken to be the object in the gaze, as indicated in figure 4B. This can be used for navigation relative landmarks (such as OBJ A).

[0081] In some embodiments, the OST device 100 is further configured to determine that a detail level in the direction of gaze in the map (such as for a detected object, OBJ A) exceeds a detail threshold level, and in response thereto deactivate the radar sensor in the direction of gaze in order to save power.

[0082] In some embodiments the detail threshold level relates to a resolution of data gathered for that area of the map. In some embodiments the detail threshold level relates to a time, wherein a newly updated portion of the map need not be scanned as thoroughly as an area not visited for a longer time.

[0083] As indicated above, the OST device 100 is - in some embodiments - configured to adapt the map based on the received radar information. In some such embodiments the OST device 100 is further configured to adapt the radar beam to a high-detail configuration in the direction of gaze and supplement data from deduced reckoning sensors 105 with data from the radar sensor to supplement the map. As a skilled person would understand, a high-detail configuration would be achieved by a high bandwidth and / or a narrow beamwidth.

[0084] In some embodiments, and as is shown in figure 4C, the OST device is configured to determine that there is no object in the map in the direction of gaze, and in response thereto add the object in the direction of gaze to the map.

[0085] Similarly, the OST device 100 is configured to determine that there is an object in the map, but that the object is stored at a detail level falling below a threshold level. In both these situations the OST device 100 may further be configured to adapt the beam of the radar sensor to a high-precision beam for receiving detailed radar information on the object in the direction of gaze to provide a detailed representation in the map. The high-precision beam can be provided by adapting any, some or all of the parameters mentioned herein. For example, by adapting the radar transmission for the beam to use a wider pulse width, higher pulse repetition frequency, or increase the bandwidth of the RF operating frequency band, or performing sharper TX beamforming in the direction of the gaze. It should be noted that even though the examples above mention transmit beams, the teachings herein may equally be applied to a receiving beam, or both. The adaptation of the operation for either of the beams, may be the same or may be different.

[0086] Figure 5 shows a general flowchart for a method according to the teachings herein. The method corresponds to the operation of the OST device 100 as discussed in the above, wherein said method comprises receiving 510 gaze tracking information from the gaze-tracking sensor, determining 520 a direction of gaze, and in response thereto adapting 530 the radar sensor based on the gazetracking information to form a radar beam in the direction of the gaze.

[0087] It should be noted that the other functionalities discussed herein may be included in some or all of the functionalities discussed in relation to figure 5 and the method of figure 5 is a general method and also allows for implementing other features as disclosed in above as sub functionality of any part of the method as disclosed.

[0088] Figure 6 shows a component view for a software component or module arrangement 600 according to some embodiments of the teachings herein. The software component arrangement 600 is adapted to be used in an OST device 100 as taught herein for providing adaptation as taught herein and corresponds to the operation of the OST device 100 in the above. The software component arrangement 600 comprises a software component 610 for receiving gaze tracking information from the gaze-tracking sensor, a software component 620 for determining a direction of gaze, and a software component 630 for adapting the radar sensor based on the gaze-tracking information to form a radar beam in the direction of the gaze in response thereto. In some embodiments the software component arrangement 600 also comprises software component(s) 640 for further functionalities as discussed in the teachings herein.

[0089] For the context of the teachings herein a software code module may be replaced or supplemented by a software component. Alternatively for the context of the teachings herein a software code module may be replaced or supplemented by a circuit configured for performing a corresponding function.

[0090] Figure 7 shows a schematic view of a computer-readable medium 102 carrying computer instructions 121 that when loaded into and executed by a controller of an OST device 100 enables the OST device the teachings herein.

[0091] The computer-readable medium 102 may be tangible such as a hard drive or a flash memory, for example a USB memory stick or a cloud server. Alternatively, the computer-readable medium 102 may be intangible such as a signal carrying the computer instructions enabling the computer instructions to be downloaded through a network connection, such as an internet connection.

[0092] In the example of figure 7, a computer-readable medium 102 is shown as being a computer disc 102 carrying computer-readable computer instructions 121, being inserted in a computer disc reader 122. The computer disc reader 122 may be part of a cloud server 123 - or other server - or the computer disc reader may be connected to a cloud server 123 - or other server. The cloud server 123 may be part of the internet or at least connected to the internet. The cloud server 123 may alternatively be connected through a proprietary or dedicated connection. In one example embodiment, the computer instructions are stored at a remote server 123 and be downloaded to the memory 102 of the OST device 100 for being executed by the controller 101.

[0093] The computer disc reader 122 may also or alternatively be connected to (or possibly inserted into) a OST device 100 for transferring the computer-readable computer instructions 121 to a controller of the OST device via a memory of the OST viewing device 100). OST device

[0094] Figure 7 shows both the situation when a OST device 100 receives the computer-readable computer instructions 121 via a server connection and the situation when another OST device 100 receives the computer-readable computer instructions 121 through a wired interface. This enables for computer-readable computer instructions 121 being downloaded into a OST viewing device 100 thereby enabling the OST device 100 to operate according to and implement the invention as disclosed herein.

Claims

CLAIMS1. An Optical-See-Through, OST, device (100) comprising a gaze-tracking sensor (103), a directional radar sensor (104) configured for steering a beam of radio waves, a memory (102) configured to store a map (M) and a controller (101), wherein the OST device is arranged to be worn on a user's head, wherein the controller (101) is configured to receive gaze tracking information from the gaze-tracking sensor, determine a direction of gaze, and in response thereto adapt the radar sensor based on the gaze-tracking information to form a radar beam in the direction of the gaze.

2. The OST device (100) according to claim 1, wherein the wherein the controller (101) is further configured to determine an object that is in the direction of gaze and in response thereto adapt parameter settings of the radar sensor.

3. The OST device (100) according to claim 2, wherein the controller (101) is further configured to determine a distance to the object that is in the direction of gaze and in response thereto adapt parameter settings of the radar sensor .

4. The OST device (100) according to claim 2 or 3, wherein the controller (101) is further configured to determine a speed of the object in the direction of gaze and in response thereto adapt parameter settings of the radar sensor.

5. The OST device (100) according to any of claims 2 to 4, further comprising an image sensor, wherein the controller (101) is further configured to determine the object in the direction of gaze by receiving image data from the image sensor and determining the object utilizing image processing techniques.

6. The OST device (100) according to any of claims 2 to 5, wherein the controller (101) is further configured to determine the object in the direction of gaze by determining a pose of the user, determining an object in the map based on the direction of gaze and the pose of the user and determining the object in the direction of gaze to be the object in the map.

7. The OST device (100) according to claim 6, wherein the controller (101) is further configured to determine a distance to the object in the direction of gaze.

8. The OST device (100) according to any of claims 2 to 7, wherein the controller (101) is further configured to determine the object in the direction of gaze by receiving radar data from the radar device 104 and determining the object utilizing radar processing techniques.

9. The OST device (100) according to any of claims 2 to 7, wherein the controller (101) is further configured to determine that there is no object in the map in the direction of gaze, and in response thereto add the object in the direction of gaze to the map.

10. The OST device (100) according to claims 7 an 9, wherein the controller (101) is further configured to determine that there is no object in the map in the direction of gaze at the determined distance,11. The OST device (100) according to any of claims 2 to 10, wherein the controller (101) is further configured to adapt the beam of the radar sensor to a high-precision beam for receiving detailed radar information on the object in the direction of gaze.

12. The OST device (100) according to any preceding claim, wherein the controller (101) is further configured to adapt the map based on the received radar information.

13. The OST device (100) according to any preceding claim, wherein the controller (101) is further configured to determine that a detail level in the direction of gaze in the map exceeds a detail threshold level, and in response thereto deactivate the radar sensor in the direction of gaze.

14. The OST device (100) according to any preceding claim, wherein the controller (101) is further configured to adapt the beam of the radar by scanning with a first power level in the direction of gaze and a second power level outside the direction of gaze, wherein the first power level is higher than the second power level.

15. The OST device (100) according to claim 14, wherein the second power level is 0.

16. The OST device (100) according to claim 14 or 15, wherein outside the direction of gaze includes directions exceeding an angular difference from the direction of gaze, wherein the angular difference is 5, 10 or 15 degrees.

17. The OST device (100) according to any preceding claim, further comprising deduced reckoning sensors (105) and wherein the controller (101) is further configured to adapt the radar beam to a high-detail configuration in the direction of gaze and supplement data from deduced reckoning sensors (105) with data from the radar sensor to supplement the map.

18. The OST device (100) according to any preceding claim, wherein the controller (101) is further configured to adapt the radar beam by selecting a configuration, wherein a configuration includes one or more settings for transmit power, pulse width, pulse repetition frequency, Radio Frequency operating frequency, sweep time, bandwidth, chirp rate, carrier frequency , radar frame, time delay between frames, transmit antennas and transmit beamwidth.

19. A method for use in an OST device (100) comprising a gaze-tracking sensor (104), a directional radar sensor (103) configured for steering a beam of radio waves, a memory (102) configured to store a map and a controller (101), wherein the OST device is arranged to be worn on a user's head, wherein the method comprises receiving gaze tracking information from the gaze-tracking sensor,determining a direction of gaze, and in response thereto adapting the radar sensor based on the gaze-tracking information to form a radar beam in the direction of the gaze.

20. A computer-readable medium (720) carrying computer instructions (721) that when loaded into and executed by a controller (101) of an OST device (100) enables the OST device (100) to implement the method according to claim 19.

21. A software component arrangement for adapting a radar beam of a OST device (100) comprising a gaze-tracking sensor (104), a directional radar sensor (103) configured for steering a beam of radio waves, a memory (102) configured to store a map and a controller (101), wherein the OST device is arranged to be worn on a user's head, wherein the software component arrangement comprises a software component for receiving gaze tracking information from the gaze-tracking sensor, a software component for determining a direction of gaze, and a software component for adapting the radar sensor based on the gaze-tracking information to form a radar beam in the direction of the gaze in response thereto.

Images