DETERMINATION OF THE POSITION AND ALIGNMENT OF A MOBILE DISPLAY USING MICRO-PULSE RADAR
Patent Information
- Application Number
- DE602015090326
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2014-03-28
- Filing Date
- 2015-02-18
- Publication Date
- 2024-11-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Conventional augmented reality systems face challenges in accurately determining the position and orientation of mobile displays relative to the environment and the user's viewpoint, leading to virtual objects appearing poorly anchored and occlusions being inaccurately represented, resulting in unnatural and 'swimming' virtual images.
The use of micropower impulse radar (MIR) technology to create a depth map of the environment and generate a viewing matrix, accurately determining the position and orientation of the display and user's viewpoint, allowing for precise placement and occlusion of virtual objects within the augmented reality image.
This approach provides accurate and natural integration of virtual objects into the real environment, preventing 'swimming' and enhancing occlusion representation, with MIR's high accuracy and low power consumption enabling real-time, interference-free operation.
Description
TECHNICAL FIELD
[0001] Embodiments described herein generally relate to the field of electronic devices and, more particularly, determination of mobile display position and orientation using micropower impulse radar.BACKGROUND
[0002] In the operation of a mobile electronic device, there may be a need for an accurate determination of the position and orientation of the device. In an example, an augmented reality image may combine images of an environment containing actual objects with images of virtual objects with the intent of providing a combined image that seamlessly merges such image components.
[0003] However, conventional systems often have difficulty in merging virtual and real images because of inaccuracy in the computed position for such elements in relation to a display being used to depict the augmented image. The inaccuracy in said computed position arises from insufficient accuracy in the measurement of position and orientation of the display being used to depict the augmented image. As a result, virtual images are often not well anchored to the environment, causing such images to often "swim" in the environment rather than naturally existing or moving in the environment.
[0004] Further, the merging of real and virtual objects requires an accurate depiction of occlusions in which a virtual object is partially hidden behind a real object. If positions and orientations are not accurately presented, the border between real objects and virtual objects will be inaccurate, which may create gaps in the augmented reality images rather than natural looking delineations between virtual and actual objects.
[0005] US 6 181 302 B1 describes a navigation system for a marine vessel that employs a composite image produced by superimposing a virtual image comprising navigational data over a real image of a portion of a surrounding environment, comprising: an ocular device images light reflected from the portion of the surrounding environment so as to directly form a field of view of the real image of the portion of the surrounding environment on a user's retina; a virtual image generator that produces an optical signal directed toward the user's retina so as to form a virtual image on the retina; an azimuth sensor that produces an azimuth signal being indicative of an orientation of the field of view formed by the ocular device; an absolute position sensing instrument that produces an absolute position signal indicative of a position of the marine vessel; and a navigational computer coupled in communication with the ocular device and the plurality of instruments, said navigational computer processes the azimuth signal and the absolute position signal and determines the field of view produced by the ocular device; determines a portion of the navigational data that should be included in the virtual image; and includes only said portion of the navigational data thus determined in the composite image.
[0006] US 2013 / 050260 describes a method for navigating concurrently and from point-to-point through multiple reality models, which includes: generating, at a processor, a first navigatable virtual view of a first location of interest, wherein the first location of interest is one of a first virtual location and a first non-virtual location; and concurrently with the generating the first navigatable virtual view of the first location of interest, generating, at the processor, a second navigatable virtual view corresponding to a current physical position of an object, such that real-time sight at the current physical position is enabled within the second navigatable virtual view.
[0007] WO 99 / 27496 describes a system for combining virtual images with a real-world scene within a field of interest for an observer, said system comprising: a range scanner scanning the field of interest and generating range data indicating the distance of real-world objects within the field of interest; a computer model simulating a virtual entity and producing a virtual image of said virtual entity at a location within the field of interest; means for generating masked virtual objects from said range data and said virtual image indicating those portions of said virtual image that are visible in the field of interest; means for combining said masked virtual objects and a realworld image of the field of interest to create a combined image in which said virtual image appears in the real-world image; and display means for displaying said combined image to the observer.
[0008] US 2012 / 274498 describes a device, comprising: a personal electronic device including a logic circuit and a user interface; and at least one micro-impulse radar operatively coupled to the logic circuit and configured to probe one or more regions near the personal electronic device; wherein the logic circuit is configured to receive or generate micro-impulse radar data including information about the probed one or more regions and, in cooperation with the user interface, at least conditionally provide information about the one or more regions to a user.SUMMARY
[0009] The present invention is defined in the appended independent claims to which reference should be made. Advantageous features are set out in the appended dependent claims.BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Embodiments described here are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements. Figure 1 illustrates an embodiment of an electronic mobile device providing determination of position and orientation using micropower impulse radar; Figure 2 illustrates a block diagram of elements of a mobile device according to an embodiment including determination of position and orientation using micropower impulse radar; Figure 3 illustrates an unclaimed micropower impulse radar component or system in accordance with the present teachings; Figure 4A illustrates generation of an augmented reality image by a mobile device according to an embodiment; Figure 4B illustrates an adjusted camera view of the environment according to an embodiment; Figure 4C illustrates the rendering of a virtual object view for use in an augmented reality image according to an embodiment; Figure 4D illustrates a virtual information view for an augmented reality image according to an embodiment; Figure 4E illustrates a compositing stack for the generation of augmented reality image according to an embodiment; Figure 4F illustrates virtual objects for the generation of an augmented reality image according to an embodiment; Figure 4G illustrates a resulting composited augmented reality view according to an embodiment; Figure 5 illustrates placement of a virtual object in an augmented reality image according to an embodiment; Figure 6 is a flow chart to illustrate an embodiment of a process for determination of device position and orientation using micropower impulse radar; and Figure 7 is an illustration of an embodiment of an electronic apparatus or system to provide for determination of apparatus position and orientation using micropower impulse radar. DETAILED DESCRIPTION
[0011] Embodiments described herein are generally directed to determination of mobile display position and orientation using micropower impulse radar.
[0012] For the purposes of this description: "Mobile device" means a tablet computer, 2-in-1 or detachable computer, smartphone, handheld computer, mobile Internet device, or other mobile electronic device that includes processing and image presentation capability.
[0013] "Micropower Impulse Radar" or "MIR" means a lower power radar system producing short electromagnetic pulses. An MIR component or system generally produces a ultra-wideband (UWB) signal using a large radio spectrum (greater than 500 MHz) in short pulses, and such component or system may be referred to as an ultra-wideband micropower impulse radar, or "UMIR". As used herein, radar systems are generally referred to as MIR systems or components, which include UMIR.
[0014] "Cover" or "case" means an outer portion of an electronic device that may enclose internal components of the device.
[0015] Electronic devices such as mobile devices may be used in augmented reality (AR) operations. Augmented reality systems render virtual objects visually superimposed upon the real world. In order for an augmented reality system that attempts to simulate actual, physical objects to appear natural to a viewer, the system is required to render the virtual objects in such a manner that the objects appear to the user to behave as real, physical objects. Among other factors, the virtual objects need to appear to be firmly anchored to the environment, and such objects must be occluded in the same manner as real objects if there are occluding portions of the real environment.
[0016] Augmented reality operation has been attempted with varying degrees of success by previous conventional systems. However, a common problem with conventional systems is that rendered virtual objects tend to "swim", or appear poorly anchored to the real environment, due to errors in sensing the position of the display relative to the environment as well as errors regarding the position of the user's dominant eye (or vantage point) in relation to the display. Further, an additional problem is that occluding features of the environment are difficult to identify and characterize, and thus conventional systems often poorly represent the occluding of virtual objects.
[0017] Conventional augmented reality systems have employed digital processing of video to address object location. However, an issue with such systems is that the systems only provide a 2D projection of the viewing frustum, where a viewing frustum is a three-dimensional space that is viewed from a particular viewpoint, such as the region viewed by a viewer through a display. Information about the actual depths in the scene is lost in the 2D projection, and it is computationally expensive to undo the projection. In addition, it is not usually possible to directly determine absolute distances or depths in the video scene without resorting to reference objects in the scene, and the use of such reference objects defeats the purpose of, for example, a self-contained mobile device used for augmented reality.
[0018] The claimed teachings provide augmented reality operation that addresses the "swimming" and occlusion problems by actively interrogating the real environment through means of radar elements (such as micropower impulse radar elements) to accurately determine the position and orientation of the display relative to the surrounding environment, and accurately determine the position and orientation of the display relative to the user's viewpoint.
[0019] Figure 1 illustrates an embodiment of an electronic mobile device providing determination of position and orientation using micropower impulse radar. The device operates by utilizing ultra-wideband micropower impulse radar to construct a depth map of the environment within the viewing frustum of the mobile display, and to generate and update a viewing matrix to accurately place the user's viewing frustum in relation to the device. The generation of the position data includes determination of: (a) The position and orientation of the plane of the mobile display relative to the environment, and (b) The position of the user's viewpoint relative to the display.
[0020] As illustrated in the top view provided in Figure 1, a mobile device 110 is utilized for an augmented reality application. The device 110 includes an imaging element 120, including but not limited to a rear facing camera, to capture an image of a real environment, where the real environment includes an actual object 175 within the camera viewing frustum 160 of the device, where Figure 1 also illustrates the user viewing frustum 162. The mobile device 110 generates and provides an augmented reality image to a user 150 on a display screen 125, wherein the augmented reality includes the addition of a virtual image 170.
[0021] However, the augmented reality image will not appear natural in its environment unless virtual objects are properly rooted to the environment and appear properly occluded by actual objects, such as object 175, that are closer to the user in terms of the virtual image. As illustrated in Figure 1, the actual object silhouette ray from the camera viewpoint 182 may not indicate an occlusion of virtual object 170 by actual object 175, but the actual object silhouette ray from the user's viewpoint 182 demonstrates that from the user's viewpoint there is an occlusion of a portion of the virtual object 170.
[0022] The mobile device includes multiple radar antennae, the antennae including multiple rear facing MIR antennae 130-132 for use in determining locations and distances of actual objects such as object 175, and may further include multiple front facing MIR antennae 134-136 for use in determining a position of the vantage point of the user 150. The antennae are used to transmit and receive ultra micropower impulse radar signal pulses. MIR provides exceptional accuracy in determining positions of objects in very small time steps, and thus is a useful and powerful technology to perform these functions.
[0023] Commonly, 3D rendering programs use 3D transformation matrices to orient virtual models within the virtual world, and to position a virtual camera within the virtual world. In augmented reality applications, the viewing frustum 160 must correspond accurately to the user's viewpoint 150 and to the position and orientation of the mobile display in the real world. While depicted in two dimensions in Figure 1, the viewing frustum has a shape that is approximately that of a four-sided pyramid with the user's viewpoint 150 at the apex of the pyramid shape.
[0024] Figure 1 further illustrates the different points of origin for the camera viewing frustum and the user frustum. The difference between the viewpoints may be corrected by pre-processing the camera video to make it appear to have been captured from the user's viewpoint. To accommodate a user 150 looking at the display 125 from off axis (i.e., not straight in front of the display), the camera viewing frustum 160 may be wider than the user viewing frustum such that the pre-processing of the camera video is to be capable of selecting the appropriate subset of the camera viewing frustum to match the user's viewing frustum. A viewing matrix to describe such a viewing frustum is continuously calculated based upon the sensing of positions of key features in the environment. The viewing matrix is then provided as a service of the mobile platform operating system to applications that wish to render virtual objects in a manner that appears well anchored to the real environment.
[0025] To handle occlusions in which a virtual object is hidden at least in part by a real object in augmented reality, 3D (three dimensional) imaging programs typically include the use of a Z- or depth-buffer. A device's MIR components are used to update a depth map to provide current and accurate depth information. A depth map gathered by the MIR can be used to pre-initialize a 3D program's Z-buffer, thus providing plausible information for occluding virtual objects by real objects. Objects in the real world that occupy a depth closer to the observer than the virtual object's depth will prevent occluded portions of the virtual object from being rendered, further enhancing the illusion that the virtual object inhabits the real world, as viewed through the mobile display. Specifically, the pre-initialized depths are required to be the distances from the user's eye-point, through each pixel of the display, to the nearest real object in the environment, as opposed to the orthogonal distance from the display plane to the objects.
[0026] MIR technology provides a direct means of interrogating the environment and determining absolute distances and angles, which provides a superior means of determining orientation and position of a display in an environment. In general, MIR components run at a speed that is at many times the 3D rendering frame rate, and thus the viewing matrix and depth map the MIR system provides are always fresh and locked to the positions of the user and the display.
[0027] Further, a MIR system is not generally subject to variations in ambient lighting conditions, as is video that passively gathers light from the scene. The radar energy used by MIR is not visible to human observers, so it can be used without interfering with the user's experience of the AR environment. This is generally not possible with video, which typically uses light spectra that is visible to human observers.
[0028] Figure 2 illustrates a block diagram of elements of an apparatus according to an embodiment including determination of position and orientation using micropower impulse radar. The apparatus such as the mobile device 210 includes a micropower impulse radar system including one or more MIR modules or sub-systems 230, wherein the mobile device further includes multiple MIR antennae, which are illustrated in Figure 2 as an antenna at each corner of a cover 212 of the mobile device 210, the antennae being antenna 220 at a first corner, antenna 222 at a second corner, antenna 224 at a third corner, and antenna 226 at a fourth corner of the cover 212 of the device. Transmit and receive antennae may be one and the same. While connections for only a first set of radar antennae are shown in Figure 2 for ease of illustration, the mobile device includes a first set of radar antennae to track object positions in first direction, such as the illustrated antennae 220, 222, 224, and 226, and a second set of radar antennae to track a position of the vantage point of user the mobile device (the dominant eye of the user of the mobile device), such as an additional set of antennae 221, 223, 225, and 227. The mobile device may request and receive information regarding which eye of the user is the user's dominant eye, such as receiving such information from the user in an initial phase of the personalization of the mobile device by the user or requesting such information when a user first utilizes an augmented reality application.
[0029] The MIR system 230 generates object position data 240 and user position data 242 that is provided to a processing unit 232. The processing unit 232 further renders virtual images, which may include the use of certain virtual object data 244, wherein the data may be stored in a memory 234. The processing unit 232 generates merged image data 246 including utilization of the updated viewing matrix and depth map, wherein the merged data is provided to a display 236 to provide the augmented reality image for the user. See, for example, the processes illustrated in Figure 6 and described below.
[0030] Figure 3 illustrates a micropower impulse radar component or system according to an unclaimed aspect. A MIR system 300 for augmented reality operation includes a noise source 302 to provide noise data to a pulse repetition generator 304 to vary the amount of time between signal pulses. The noise source is provided for the purpose of allowing multiple MIR radars to operate in the same space at the same time. The variable time between pulses, together with the extremely low duty-cycle (pulse width compared to pulse repetition rate) makes it extremely unlikely that a pulse from one MIR radar would be received and mistaken for a pulse sent from a second MIR radar operating in the same space. Therefore, MIR radar units have the desirable feature that they do not interfere with one another even when multiple units are simultaneously operating in the same space, such as, for example, multiple MIR units operating simultaneously in a mobile device. The pulse repetition generator provides a pulse signal to an impulse generator and to a delay 310, the delay including a range control 312.
[0031] The impulse generator 306 generates impulses for transmission via send / receiver antenna 308. A send antenna 320 produces a transmitted signal 330, which is reflected as a return signal 332 upon impacting an object. The return signal is received at a receive antenna 322 (which may be the same unit as the send antenna 320). The received signal is provided to an impulse receiver 316, which also receives a delayed pulse from the delay 310 as controlled by a range gate 314. The delay time is swept to take samples with an impulse receiver 316 at different times, and thus distances from the antennae, resulting in a record of pulses correlated to object distances. The impulse receiver 316 provides a resulting signal to a processor 318 to process the received signals to generate one or both of object position data and user position data. The processor utilizes the object position data and virtual object data based upon a viewpoint of a user in the generation of images for an augmented reality display. During rendering of a virtual object, the z-buffer causes appropriate parts of the virtual object to be occluded and thus are not generated, which occurs prior to the alpha compositing operations, which merge camera and virtual image data. The alpha-compositing operations do not need to be aware of the z-buffer. However, the images that are generated match up correctly with the actual environment due to their accurate registration during the 3D rendering process.
[0032] As structured, a MIR system 300 can be very small in size and consume little power, and thus may be utilized in a mobile device having limited space and power capacity. The generated MIR impulses are very short compared to the repetition rate, even if said repetition rate is several Megahertz. For this reason, many MIR systems can be operating simultaneously in the same space, wherein the impulses from one MIR are not time-correlated with impulses from another because of the noise generation in the MIR system. Therefore, a MIR system 300 with the architecture illustrated in Figure 3 has great immunity to extraneous signals, including impulses from multiple MIR components operating in the same space. This property allows several MIR components to operate in concert, operating simultaneously, in a mobile device without significant interference. It is further noted that MIR technology uses extremely low power levels, and thus the human exposure to the signal output from a device is generally a fraction of the RF power employed by a typical cellular phone.
[0033] A mobile device such as a tablet computer includes multiple mounted MIR antennae in a spread pattern, such as an antenna mounted at each corner of the cover of the mobile device. In this example, four antennae may be facing the user (from the front of the device) and four antennae may be facing away from the user (from the rear of the device). From each such antenna it is possible to record time-of-flight for an impulse signal that is reflected. Each discrete time of flight of the radar impulse represents a surface of a sphere with center at a particular MIR that transmitted it. In this example, by combining measurements from each corner antenna, the intersection of four spheres of radius determined by the time of flight at each antenna is obtained. An object at a particular distance and heading from any three antennas facing in the same direction (toward or away from the user) will have returns in the three spheres that intersect at the point in space occupied by the object. Thus, by examining returns in appropriate radius spheres and by the use of well-known trigonometric algorithms applied to these measurements, radial distances from the user's eyepoint to objects in the environment can be computed and used to initialize a z-buffer.
[0034] By making an assumption that objects appear stationary relative to a tablet computer at the rate at which measurements are made, and by discarding objects or features in the environment that do not remain in static spatial relationships to one another (or in other words, by discarding objects that move), it is thus possible to accurately determine the position and orientation of the tablet computer relative to the environment sampled by the array of MIRs.
[0035] Figure 4A illustrates generation of an augmented reality image by a mobile device according to an embodiment. In Figure 4, a mobile device 410, such as a tablet computer, is utilized for augmented reality in a natural setting. As illustrated, a virtual image 420 (the dinosaur) is superimposed upon the real environment in the viewing screen 425.
[0036] In order to portray a virtual object such as the dinosaur image 420 in Figure 4 realistically for a viewer, it is necessary that the position of the virtual object does not appear to move unnaturally relative to the environment, such as, in this example, moving unnaturally relative to the ground. Although the dinosaur virtual image itself may appear to move in the scene, the feet of the dinosaur should not appear to move relative to the ground for as long as the feet are in contact with the ground. In addition, a portion of the virtual object, the tail of the dinosaur image 420, is partially occluded by a real object, a tree. In order to provide a realistic and natural augmented reality image, it is necessary that the intersection of the tree and the tail be aligned accurately relative to the user's viewpoint such that the portion of the virtual object that is occluded appears to disappear at the precise location (sometimes called the silhouette edge) where the tree begins.
[0037] The position of a virtual object depends upon the numerical values in the viewing matrix, which in turn depends upon the shape and orientation of the viewing frustum. A viewing matrix is produced using MIR data with values that precisely position the virtual object in a virtual world that is superimposed upon the real world everywhere within the viewing frustum defined by the user's viewpoint (more specifically, the user's dominant eye), and the edges of the display area of the tablet.
[0038] Figures 4B through 4G illustrate the generation and superimposition of images to create the augmented reality image provided in Figure 4A according to an embodiment: Figure 4B illustrates an adjusted camera view of the environment, showing real objects 410 such as the illustrated trees of the environment. The adjusted camera view is generated by capturing the actual environment and adjusting the view of the environment such the view matches the view of the dominant eye of the user. This may be seen as the adjustment of the camera viewing frustum 160 to match the current user viewing frustum 162, as illustrated in Figure 1.
[0039] Figure 4C illustrates the rendering of a virtual object view for use in an augmented reality image. A virtual image 420 is rendered for augmented reality imaging based on current position information, wherein the rendering of the image excludes image portions that would be occluded by real objects. In this example, based on the virtual distance of the image and the distances of the actual objects as determined by MIR, a portion of the virtual image is not rendered. The portion that is not rendered is located at the determined actual object silhouette from the user's viewpoint, such as defined by the actual object silhouette ray from the user's viewpoint 180 illustrated in Figure 1.
[0040] Figure 4D illustrates a virtual information view for an augmented reality image. The rendering of a 3D image further includes the rendering of an information view 430 that provides information regarding one or more virtual objects that are contained in the virtual view of Figure 4C. The virtual view may appear only when the virtual object 420 illustrated in Figure 4C is wholly or mostly within the user's viewing frustum. It may not be necessary for the information view 430 to be anchored to the environment in the same manner as virtual object 420, and thus the information view 430 may swim in relation to the actual objects in a view, and may, for example, be rendered to be in front of any actual objects in an augmented image such that the image is not occluded by the actual objects.
[0041] Figure 4E illustrates a compositing stack for the generation of augmented reality image. As illustrated in Figure 4E, a compositing stack for the generation of the augmented reality image for the view of the dominant eye of the user 450 includes a first layer providing adjusted camera video containing real objects 410 (as shown in Figure 4B), which is overlaid with a second layer containing the virtual 3D dinosaur virtual object 420 (as shown in Figure 4C), which is overlaid with a third layer containing the information view virtual object 430.
[0042] Figure 4F illustrates virtual objects for the generation of an augmented reality image. As shown, a combined image includes the virtual image 420 overlaid with the information view 430.
[0043] Figure 4G illustrates a resulting composited augmented reality view. As shown, the real objects 410 of the adjusted video camera view are overlaid with the virtual image 420 and the information view 430. With the adjusted camera view of the real environment the virtual image 420 is rendered to precisely match the silhouette boundary of the real objects 410, and the virtual image 420 is rendered to tie the virtual image 420 to the environment such that the virtual image does not appear to swim in the real environment.
[0044] Figure 5 illustrates placement of a virtual object in an augmented reality image. A mobile device utilizes multiple MIR elements 520 (which may include individual MIR sub-systems) to transmit radar signals 530 and to receive return signals 532. As illustrated in Figure 5, a viewing frustum 560 is from a perspective behind and to the right of the virtual object 570 (a dinosaur image), from the mobile device user's vantage point 550 (the dominant eye of the mobile device user). From the vantage point of a camera capturing Figure 5, if the virtual object 570 actually existed in the real world, the occluded portion of the tail would be seen, but such portion of virtual object 570 is occluded by the real object 575 (tree) from the mobile device user's view.
[0045] By means of the multiple antennae (four antennae in the Figure 5 illustration) that face the user (wavefronts moving towards the mobile device user in the figure), the position of the user's dominant eye 550 can be accurately determined. The position 550 is the apex of the viewing frustum 560 illustrated in Figure 5. Knowing the apex point, and knowing the precise position and orientation of the mobile device relative to the environment, (obtained by means of the four antennae and the wavefronts moving away from the user) a viewing matrix is generated that defines the viewing frustum 560 in the virtual world in alignment with the physical world as seen at an instant in time by the mobile device user, with the viewing matrix being made available to a 3D rendering program that is drawing the dinosaur virtual object 570.
[0046] In order to occlude the appropriate portion (the tail, in this example,) of virtual objects, the virtual objects in the environment are rendered using a depth map that has been pre-initialized with a depth map. The initialization of the depth map may be captured by the MIR components, wherein the depth of each point in the video raster is measured by the MIRs, or using a 3D camera component.
[0047] Figure 6 is a flow chart to illustrate an embodiment of a process for determination of device position and orientation using micropower impulse radar. Upon operation of an electronic device being enabled 600 and the device entering an augmented reality application 602, the MIR system of the electronic device may be enabled 604 (if the MIR components or sub-systems are not currently enabled) and an image depth map and viewing matrix may be initialized (where such initialization may be based on MIR operation, video camera operation, or other similar operation). The augmented reality application may include a request for identification of the user's dominant eye if this information has not previously been received.
[0048] The processing may include both a radar data process 610 and a 3D image generation process 620. The radar data process 610 may include transmitting radar signals from multiple radar antennae of the electronic device 612, where such transmission includes transmission of signals both towards the real environment for determination of the display's position and orientation and towards the user for determination of the user's vantage point. Return signals are received at the MIR antennae 614, the return signals providing reflection from actual objects in the environment and from the user. The process further includes determining display position data and user position data 616 and the updating of the depth map and viewing matrix 618, which define the depth of objects in the environment and provide an accurate location and orientation for the mobile device in relation to the actual objects in the environment and the position of the user in relation to the mobile device. In some embodiments, the generated data may be stored, as indicated by position data 632, for use in generation of augmented reality images. In some embodiments, the radar data process 610 operates quickly, and thus current position data 630 is available when needed for generation of images.
[0049] In some embodiments, the 3D image generation process 620 may include rendering of virtual image objects 622, which may include obtaining the position data 630 and certain virtual image data from memory 632. In some embodiments, the process further includes obtaining digital camera video data 634 and processing the video data to superimpose the camera data and the user's dominant eye point 624 to generate an adjusted camera video for the user's viewpoint.
[0050] In some embodiments, the process includes generation of merged image data 626 by layering of the virtual image with the adjusted camera view, wherein the merged image data utilizes the image depth map and viewing matrix to accurately position virtual images and provide accurate occlusion of the virtual images, resulting in a natural placement of the virtual image in the merged image data according to the current viewing frustum of the user. In some embodiments, the process includes displaying an augmented reality image based upon the merged image data 628.
[0051] In some embodiments, if there is no exit from the augmented reality application, the radar data process 610 and the 3D image generation process 620 continue. If there is an exit from the augmented reality application 640, then the MIR system or systems may be disabled 642 (if appropriate) and process ended 644.
[0052] Figure 7 is an illustration of an embodiment of an electronic apparatus or system to provide for determination of display position and orientation using micropower impulse radar. In this illustration, certain standard and well-known components that are not germane to the present description are not shown. Elements shown as separate elements may be combined, including, for example, an SoC (System on Chip) combining multiple elements on a single chip. The apparatus or system may include, but is not limited to, a mobile device such as a tablet computer.
[0053] The apparatus or system 700 (referred to generally herein as an apparatus) includes a cover 710. The apparatus further includes multiple radar antennae, including a first radar antenna 720, a second radar antenna 722, a third radar antenna 724, and a fourth radar antenna 726 for the transmission and reception of video images. While connections for only a first set of antennae are shown for ease of illustration, the mobile device includes a first set of antennae to track object positions in first direction, such as the illustrated antennae 720, 722, 724, and 726, and a second set of antennae to track a position of the vantage point of a user of the mobile device (the dominant eye of the user of the mobile device) in a second direction, such as an additional set of antennae 721, 723, 725, and 727. The radar antennae may be spread spatially to allow for separation of the timing of each impulse signal, such as, for example, the location of an antenna in each corner of the apparatus 700. Each of the antennae is in a different location in the apparatus. The antennae 720, 722, 724, 726 transmit and receive radar data a plurality of MIR components or sub-systems 730. The antennae are directed towards an environment for determination of a position and orientation of the apparatus 700 with respect to the objects in the environment. In some embodiments, there is an additional set of antennae 721, 723, 725, 727 directed in an opposite direction (towards a user of the apparatus) to determine the position of the vantage point of the user in relation to the apparatus 700.
[0054] The apparatus 700 includes an interconnect 740 or other communication means for transmission of data, the MIR components 730 being coupled with the interconnect 740. The interconnect 740 is illustrated as a single interconnect for simplicity, but may represent multiple different interconnects or buses and the component connections to such interconnects may vary. The interconnect 740 shown in Figure 7 is an abstraction that represents any one or more separate physical buses, point-to-point connections, or both connected by appropriate bridges, adapters, or controllers.
[0055] The apparatus 700 may include a processing means such as one or more processors 742 coupled to the interconnect 740 for processing information. The processors 742 may comprise one or more physical processors and one or more logical processors. In some embodiments, the processors may include one or more general-purpose processors or special- processor processors. In some embodiments, the processors 742 operate to process radar signal data to generate data describing the position and orientation of the apparatus in relation to real objects in an environment, the vantage point of a user of the apparatus 700, or both.
[0056] The apparatus includes one or more image capture elements 764, such as a camera, which may include the capture of images of an environment for generation of augmented reality images. The image capture elements include a 3D camera, wherein the 3D camera may be utilized in the initialization of position data for the apparatus 700.
[0057] In some embodiments, the apparatus 700 further comprises a random access memory (RAM) or other dynamic storage device or element as a main memory 744 for storing information and instructions to be executed by the processors 742. The apparatus 700 may include one or more non-volatile memory elements 746, including, for example, flash memory, for the storage of certain elements. The apparatus 700 also may comprise a read only memory (ROM) 748 or other static storage device for storing static information and instructions for the processors 742, and data storage 750, such as a solid state drive, for the storage of data. Memory of the apparatus 700 may include storage of data related to the determination of position data, such as which of the user's eyes is dominant, and the generation of image data for the presentation of augmented reality images.
[0058] The apparatus 700 includes one or more transmitters or receivers 752 coupled to the interconnect 740. The apparatus 700 may include one or more antennae 756 for the transmission and reception of data via wireless communication, where the antennae may include bipole and monopole antennae, and one or more ports 754 for the transmission and reception of data via wired communications.
[0059] The apparatus 700 includes one or more input devices 758 for the input of data, including hard and soft buttons, ajoy stick, a mouse or other pointing device, voice command system, or gesture recognition system.
[0060] The apparatus 700 includes an output display 760, where the display 760 may include a liquid crystal display (LCD) or any other display technology, for displaying information or content to a user. In some environments, the display 760 may include a touch-screen that is also utilized as at least a part of an input device 758. The output display 760 may be utilized in the display of augmented reality images.
[0061] The apparatus 700 may also comprise a battery or other power source 762, which may include a solar cell, a fuel cell, a charged capacitor, near field inductive coupling, or other system or device for providing or generating power in the apparatus 700. The power provided by the power source 762 may be distributed as required to elements of the apparatus 700.
[0062] In the description above, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the claimed invention. It will be apparent, however, to one skilled in the art that the inventions may be practiced without some of these specific details. In other instances, well-known structures and devices are shown in block diagram form. There may be intermediate structure between illustrated components. The components described or illustrated herein may have additional inputs or outputs that are not illustrated or described.
[0063] The claimed invention includes various processes. These processes may be performed by hardware components or may be embodied in computer program or machine- executable instructions, which may be used to cause a general-purpose or special-purpose processor or logic circuits programmed with the instructions to perform the processes. Alternatively, the processes may be performed by a combination of hardware and software.
[0064] Portions of the claimed invention may be provided as a computer program product, which may include a computer-readable medium having stored thereon computer program instructions, which may be used to program a computer (or other electronic devices) for execution by one or more processors to perform a process. The computer-readable medium may include, but is not limited to, magnetic disks, optical disks, compact disk read-only memory (CD-ROM), and magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically-erasable programmable read-only memory (EEPROM), magnet or optical cards, flash memory, or other type of computer-readable medium suitable for storing electronic instructions. Moreover, embodiments may also be downloaded as a computer program product, wherein the program may be transferred from a remote computer to a requesting computer.
Claims
1. A mobile device (110, 210) comprising a camera; a display (236) to present images captured with the camera; radar components (230) to generate a plurality of radar signal pulses and to generate distance data based on received return signals; a first plurality of radar antennae (220, 222, 224, 226) to transmit the plurality of radar signal pulses in a rear facing direction of the apparatus and a second plurality of radar antennae (221, 223, 225, 227) to transmit the plurality of radar signal pulses in a front facing direction of the apparatus and to receive the return signals; and a processor (232) to process signals and data, wherein the processor is to: process the return signals received by the first and second plurality of radar antennae to determine a position and orientation of the display with respect to real objects in an environment and to determine a position of a vantage point of a user of the apparatus, and generate an augmented image including rendering a virtual object and superimposing the virtual object on an image including one or more real objects, the rendering of the virtual image being based at least in part on the determined position and orientation of the display and the determined vantage point of the user of the apparatus; wherein processing the return signals includes updating a depth map providing distances of real objects and updating a viewing matrix, the viewing matrix defining a viewing frustum of a user of the apparatus.
2. The mobile device (110, 210) of claim 1, wherein the radar components (230) are micropower impulse radar, MIR, components.
3. The mobile device (110, 210) of claim 1, wherein rendering the virtual image excludes a portion of the virtual image that is occluded by a first real object, the occlusion being based on the depth map and the viewing matrix.
4. The mobile device (110, 210) of claim 1, wherein the processor (232) is to preprocess the image of the environment to appear to have been captured from the vantage point of the user.
5. The mobile device (110, 210) of claim 1, wherein the apparatus is a mobile device.
6. The mobile device (110, 210) of claim 4, wherein each of the plurality of radar antennae is in a different location in the mobile device.
7. A method of operating a mobile device (110, 210) comprising: capturing an image of an environment with a camera of the mobile device; transmitting (612) radar signal pulses from a plurality of radar antennae in the mobile device comprising: transmitting radar signal pulses from a first plurality of radar antennae in the mobile device in a rear facing direction of the mobile device; transmitting radar signal pulses from a second plurality of radar antennae in the mobile device in a front facing direction of the mobile device; receiving (614) return signals at the first and second plurality of radar antenna; processing (630) the return signals received at the plurality of radar antennae to determine a position and orientation of a display with respect to real objects in the environment and a position of a vantage point of a user of the mobile device; rendering (622) a three-dimensional, 3D, virtual object, wherein the rendering of the virtual object is based at least in part on the determined position and orientation of the display and the determined position of the vantage point of the user; and generating (626) an augmented reality image by superimposing the virtual object on the image of the environment; wherein processing the return signals received at the plurality of radar antennae includes updating a depth map providing distances of objects in the environment and updating a viewing matrix, the viewing matrix defining a viewing frustum of the user of the mobile device.
8. The method of claim 7, wherein the radar signal pulses are transmitted using a micropower impulse radar, MIR, system.
9. The method of claim 7, further comprising preprocessing the image of the environment to appear to have been captured from the vantage point of the user.
