Immersive Digital Map Navigation Using One Hand

Abstract

To localize the user within a digital map using an immersive view at the user's location, the client device determines the position and orientation of the camera view of the client device. The client device then presents an immersive view of the three-dimensional (3D) map from a parse of a virtual camera having an orientation and position that match the orientation and position of the camera view within the client device.

Description

Technical Field

[0001] The present disclosure relates to interactive geographic applications, and more particularly to providing an immersive view of a three-dimensional (3D) map from the same position and / or orientation as the user to locate the user within a map display.

Background Art

[0002] For the purpose of generally presenting the context of the present disclosure, the background art provided herein is described. The work of the inventors specified herein is not admitted as prior art to the present disclosure, either explicitly or implicitly, to the extent described in this background art section and also in aspects of the description that may not be eligible as prior art at the time of filing.

[0003] Various computing devices support geographic software applications for displaying interactive digital maps of geographic areas. Many map applications provide the user with the ability not only to select the type of map information or features to view, but also to adjust the display of the digital map. For example, the user may select from several scales and map viewing modes, such as a basic map mode that presents a conventional road map view, a satellite mode, a street level mode, or a three-dimensional (3D) view.

[0004] Typically, when a user launches a mapping application, the mapping application may present a window that defines a visible portion of the digital map, which can be referred to as a "viewport" of the geographic area that includes the user's current location. However, it can be difficult for the user to locate themselves within the viewport, for example, to determine where they are relative to a building or point of interest (POI).

[0005] Furthermore, mapping applications typically provide controls for direction, rotation, and zoom to position the viewport over a desired location. For example, these controls can be provided in the form of buttons overlaid on a digital map. As another example, a mapping application operating on a device with a touch screen can support user gestures, so that the user can pan across the digital map by swiping their finger in a desired direction, zoom in on an area by pinching two fingers together, zoom out of an area by separating two fingers, rotate the digital map by rotating one finger while holding the other finger in place, and tilt the digital map by sliding two fingers in a desired tilt direction, among other things. SUMMARY OF THE INVENTION

[0006] To localize the user within a digital map, the user can orient the camera view of the client device in the direction the user is facing. The client device can then determine its position and orientation based on sensor data from sensors in the client device, such as a global positioning system (GPS), compass, gyroscope, accelerometer, transceiver, camera, etc. The client device then presents an immersive view 3D map from a perspective of a virtual camera having the same orientation and position as the camera view of the client device.

[0007] In some embodiments, the client device may determine an initial zoom level of the immersive view based on the distance between the user and the client device. For example, the client device can identify the user's face in the front camera view and determine the distance between the user and the client device based on the size of the user's face in the camera view. If the user's face fills the entire camera view, the client device can determine that the user is closer to the client device, for example, than if the user's face occupies only a small portion of the camera view.

[0008] By presenting an immersive view of a 3D map based on the orientation and / or position of a client device, a user may be able to orient themselves within the map and identify where they are relative to a particular building or point of interest (POI). The user can then interact with buildings or POIs within the 3D map, for example, by selecting an icon to overlay on the POI to view additional information about the POI. Further, the user can pan or zoom the 3D map to view locations beyond the horizon and interact with those locations.

[0009] Furthermore, the immersive view can be presented without the need to determine the position and / or orientation at the level of accuracy required by alternative systems such as augmented reality systems. For example, in an augmented reality system, it is necessary to determine the position and / or orientation of the client device very precisely in order to overlay augmented reality icons on real-world objects. In contrast, the immersive view of a 3D map can be from the perspective of a virtual camera that is slightly offset from the position and / or orientation of the client device's camera view, yet the user may still be able to orient themselves within the 3D map and interact with the 3D map. In this way, the immersive view can be presented more frequently than alternative systems, and the immersive view can be generated without requiring as much processing power as would be needed to determine the user's precise position and / or orientation.

[0010] In some embodiments, the user may also select user controls to view a two-dimensional (2D) map having a specific viewport. Without using finger gestures (which typically require holding the phone with one hand and performing gestures with the other hand), to reposition the viewport of the 2D map or the immersive view of the 3D map, the client device identifies the surface of the virtual plane of the 2D map or the surface of the virtual sphere surrounding the user of the 3D map. For example, the surface of the virtual plane may be located behind and parallel to the surface of the user interface on the client device. In other examples, the surface of the virtual plane may be tilted with respect to the surface of the user interface, as described in more detail below. In yet other examples, the surface of the virtual plane may be aligned with the ground. The surface of the virtual sphere may be a 360-degree view around the user, and the surface is at a fixed distance from the user. In any case, the client device detects the movement of the client device within a specific direction with respect to the surface of the virtual plane or the virtual sphere. Next, the client device re-positions the viewport of the digital map according to the direction of movement of the client device.

[0011] In this way, the user can re-position the viewport by holding the client device with one hand and moving it in a specific direction. The client device can re-position the viewport by moving its entire body from one position to another, and the user can also re-position the viewport by only moving their hand while staying in place or within a threshold region (e.g., within a radius of 5 meters, a radius of 10 meters, etc.). Therefore, the client device does not need to use a positioning sensor such as a global positioning system (GPS) sensor to detect the movement of the client device that may have inaccuracies. Instead, the client device uses sensors within the client device, such as an accelerometer, a gyroscope, an inertial measurement unit (IMU), a compass, etc., to detect the movement.

[0012] For example, when the viewport of a 2D map is positioned such that north is on top of the digital map, if the user moves the client device upward relative to the surface of the virtual plane, the viewport pans the digital map north. If the user moves the client device downward relative to the surface of the virtual plane, the viewport pans the digital map south. If the user moves the client device leftward relative to the virtual plane, the viewport pans the digital map west, and if the user moves the client device rightward relative to the virtual plane, the viewport pans the digital map east. Further, if the user moves the client device inward and closer to the surface of the virtual plane, the viewport zooms in on the area depicted within the digital map, and if the user moves the client device outward and away from the surface of the virtual plane, the viewport zooms out on the area depicted within the digital map. Also, the user may rotate or tilt the viewport by rotating or tilting the client device.

[0013] In another example, in an immersive view, when the user moves the client device upward relative to the surface of the virtual sphere, the viewport pans upward, such as to the sky, above a building, etc. When the user moves the client device downward relative to the surface of the virtual sphere, the viewport pans downward, such as towards the ground in the immersive view. When the user moves the client device to the left relative to the virtual sphere and the user is facing north, the viewport pans the immersive view northwest, and when the user moves the client device to the right relative to the virtual sphere, the viewport pans the immersive view northeast. Further, when the user moves the client device inward towards the surface of the virtual sphere to get closer, the viewport zooms in on the area depicted within the immersive view, and when the user moves the client device outward away from the surface of the virtual sphere, the viewport zooms out on the area depicted within the immersive view. The user may pan the client device, for example, to focus on a particular building within the immersive view. The user may then move the client device inward to view the particular building at a larger scale.

[0014] In some embodiments, when the user moves the client device, the user may provide a user control to lock the position of the viewport or the position of the virtual camera for the immersive view. For example, the user may press a finger on the user interface and hold that finger on the user interface to maintain the viewport or the immersive view in a locked position (e.g., a long-press gesture) such that the position of the viewport or the immersive view does not change because the user's finger remains in contact with the user interface. Then when the user releases their finger, the position of the viewport or the virtual camera for the immersive view may be unlocked and may continue to change in accordance with the movement of the client device.

[0015] In this way, the user can move the client device without panning, zooming, rotating, or tilting the viewport or immersive view. For example, the user can move the client device to the right to pan the digital map eastward. If the user wants to view a further eastern region but cannot move their hand further to the right, the user can lock the position of the viewport and move their hand back to the left with respect to their body. The user can then release the locked position and continue to move their hand to the right to pan further eastward. Advantageously, this enables the user to easily pan within the digital map while minimizing the movement of the user's hand and / or body. In other embodiments, the viewport or immersive view may be defaulted to the locked position. The user can unlock the position of the viewport or immersive view by long-pressing on the user interface and moving the client device while keeping a finger on the user interface.

[0016] The mapping application improves the user experience and makes it more convenient to operate the digital map display by repositioning the viewport of the digital map without the need to use finger gestures. The user does not need to use both hands, whereby one of the user's hands is freed up for additional activities such as holding or carrying other objects. Furthermore, the mapping application improves driver distraction by reducing the amount of concentration required to operate the digital map display. For example, if a driver is searching in multiple directions on a digital map, the driver does not need to take their hand off the steering wheel or divert their concentration from the road to perform multi-finger gestures to operate the digital map. Still further, the mapping application improves usability for people with disabilities, arthritis, or other neurological or neuromuscular disorders that affect the ability to control fingers or use both hands.

[0017] One embodiment of these techniques is a method for localizing a user within a digital map using an immersive view at the user's location. The method includes determining the location of a client device and determining the orientation of the client device. Further, the method includes presenting an immersive view of a three-dimensional (3D) map from a perspective of a virtual camera having an orientation and a position that match the orientation and the position of the camera view within the client device.

[0018] Another embodiment of these techniques is a client device for localizing a user within a digital map using an immersive view at the user's location. The client device includes a user interface, one or more processors coupled to the user interface, and a computer-readable medium storing instructions. The computer-readable medium can be non-transitory. The instructions, when executed on the one or more processors, cause the client device to determine the location of the client device and determine the orientation of the client device. Further, these instructions cause the client device to present, via the user interface, an immersive view of a three-dimensional (3D) map from a perspective of a virtual camera having an orientation and a position that match the orientation and the position of the camera view within the client device.

[0019] Another embodiment of these techniques is a method for generating an immersive view at the user's location for localizing a user within a digital map. The method includes obtaining sensor data from one or more sensors within a client device and determining the orientation and the location of the client device based on the sensor data. Further, the method includes generating an immersive view of a three-dimensional (3D) map from a perspective of a virtual camera having an orientation and a position that match the orientation and the position of the camera view within the client device and providing the immersive view for display via the client device.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

[0021] Overview Generally speaking, techniques for localizing a user within a digital map can be implemented in a system that includes one or several client devices, one or several network servers, or a combination of these. However, for clarity, the following examples mainly focus on embodiments where the user positions the camera view of the client device to face the direction the user is facing. The mapping application on the client device obtains sensor data from sensors within the client device, such as positioning sensors (e.g., Global Positioning System (GPS) sensors), compasses, gyroscopes, accelerometers, magnetometers, transceivers that receive wireless signals from nearby devices, rear cameras that capture images of the current rear camera view, front cameras that capture images of the current front camera view, or any other suitable sensors within the client device. Next, the mapping application can analyze the sensor data to determine the position and orientation of the client device, and more specifically, the orientation of the rear camera view of the client device. Next, the mapping application sends a request to the server device to obtain an immersive view of the 3D map from the same perspective as the client device. In other embodiments, the mapping application sends the sensor data to the server device, and the server device analyzes the sensor data to determine the position and orientation of the client device. In yet other embodiments, the client device does not obtain sensor data from the camera, and the rear camera pose of the client device is determined using other sensor data from other sensors within the client device.

[0022] In any case, the server device can generate an immersive view of the 3D map by acquiring 3D map data from a map database having a virtual camera pose that matches the rear camera pose of the client device. The server device may also select an initial zoom level of the immersive view based on the distance from the user to the front camera. For example, the client device may determine the distance from the user to the front camera based on the size of the user's face in the front camera view by analyzing an image of the current front camera view. In other embodiments, the client device may send an image of the current front camera view to the server device, and the server device may determine the distance from the user to the front camera by analyzing the image of the current front camera view.

[0023] In any case, the server device then sends the immersive view of the 3D map and / or the 3D map data of the immersive view to the client device for display via a mapping application. In some embodiments, the server device sends additional immersive views at different zoom levels or orientations. In this way, the user can pan or zoom the 3D map without the mapping application making additional server requests. The client device may then present a 3D map representing the user's current position and orientation and an immersive view for the user to interact with.

[0024] As used herein, the term "pose" may refer to the position and / or orientation of an object. For example, the pose of a camera may be a combination of the camera's current position and the direction the camera is facing, the pose of the camera may be the camera's current position, or the pose of the camera may be the direction the camera is facing.

[0025] Exemplary Hardware and Software Components An exemplary communication system 100 capable of implementing a map display positioning system is shown in FIG. 1. The communication system 100 includes a client device 10 configured to execute a geographic application 122, which can also be referred to as a "mapping application 122". Depending on the implementation, the application 122 can display an interactive digital map, request and receive routing data, provide other navigation directions including driving directions, walking directions, or audio navigation directions, provide various location-specific content, and so on. The user can operate the client device 10 to display a 2D or 3D digital map while navigating to various locations.

[0026] In addition to the client device 10, the communication system 100 includes a server device 60 configured to provide an immersive view and / or a viewport to the client device 10. In an exemplary embodiment, the server device 60 can be communicatively coupled to a database 80 that stores real-world images of various geographic regions. In this way, the server device 60 can identify the user's pose by comparing the real-world image from the camera view of the client device 10 with the real-world images in the database 80. The database 80 may also store 3D map data for generating an immersive view from a specific virtual camera pose.

[0027] More generally, the server device 60 can communicate with one or several databases storing any type of appropriate geospatial information, or information that can be linked to a geographical context. The communication system 100 can also include, for example, a navigation data server 34 that provides navigation directions such as driving direction, walking direction, cycling direction, or public transportation direction. Further, the communication system 100 can include a map data server 50 that provides 2D or 3D map data to the server device 60 to generate a map display. Devices operating within the communication system 100 can be interconnected via the communication network 30.

[0028] In various embodiments, the client device 10 can be a smartphone or a tablet computer. The client device 10 can include a memory 120, one or more processors (CPUs) 116, a graphics processing unit (GPU) 112, an I / O module 114 including a microphone and a speaker, a user interface (UI) 32, and one or several sensors 19 including a global positioning system (GPS) module, a compass, a gyroscope, an accelerometer, a magnetometer, a camera, a transceiver, etc. The memory 120 can be a non-transitory memory and can include one or several appropriate memory modules such as random access memory (RAM), read-only memory (ROM), flash memory, other types of persistent memory, etc. The I / O module 114 can be, for example, a touch screen. In various embodiments, the client device 10 can include fewer components than those shown in FIG. 1, or conversely, additional components. In other embodiments, the client device 10 can be any appropriate portable or non-portable computing device. For example, the client device 10 can be a laptop computer, a desktop computer, a wearable device such as a smartwatch or smart glasses, a virtual reality headset, etc.

[0029] The memory 120 stores an operating system (OS) 126, which can be any suitable mobile or general-purpose operating system of any type. The OS 126 can include an application programming interface (API) function that enables an application to obtain sensor readings. For example, a software application configured to execute on the computing device 10 can include instructions to call the OS 126 API to obtain the current position of the client device 10 at that instant. The API can also return a quantitative (e.g., as a percentage) indication of how certain the API is of the estimated value.

[0030] The memory 120 also stores a mapping application 122, which is configured to generate interactive 2D and 3D digital maps and / or perform other geographic functions as described above. The mapping application 122 can receive navigation instructions, including audio navigation instructions, and present the navigation instructions. The mapping application 122 can also display a driving direction, a walking direction, or a transportation direction, and generally provide functions related to geography, geographical location, navigation, etc.

[0031] The mapping application 122 can include an immersive view controller 134 that generates a request for an immersive view of a 3D map and transmits the request to the server device 60. The request can include an indication of the pose of the rear camera view of the client device 10, or sensor data indicating the pose of the rear camera view of the client device 10, such as a real-world image from the rear camera, GPS data, accelerometer data, gyroscope data, compass data, magnetometer data, transceiver data indicating the signal strength of a signal received from a nearby wireless device, etc.

[0032] The request may also include an indication of the distance between the user and the front camera view of the client device 10, or sensor data indicating the distance between the user and the front camera view of the client device 10, for example, including a real-world image from the front camera. Next, the server device 60 may analyze the real-world image from the front camera to identify the user within the front camera view, for example, using face detection techniques. Next, the server device 60 may determine the distance from the user to the client device 10, for example, based on the size of the user's face within the front camera view.

[0033] Next, the server device 60 may determine the pose of the virtual camera of the immersive view based on the pose of the rear camera view of the client device 10. The server device 60 may also determine the initial zoom level of the immersive view based on the distance from the user to the client device 10. Next, the server device 60 may obtain an immersive view having a matching virtual camera pose and zoom level from the database 80 and provide the immersive view to the immersive view controller 134 for display.

[0034] In addition to requesting and displaying an immersive view, the immersive view controller 134 receives requests from the user to pan or zoom an immersive view or viewport of a 2D map display, for example, when the user selects a user control to view a 2D map display. The immersive view controller 134 may generate a virtual sphere surrounding the user, as will be described in more detail below with reference to FIG. 2. For example, the virtual sphere may have a radius of 1 meter surrounding the user.

[0035] Next, when the user moves the client device 10 in a specific direction with respect to the virtual sphere, the immersive view controller 134 detects the movement based on sensor data from the sensor 19 of the client device 10, such as an accelerometer or a gyroscope, and pans the 3D map in the movement direction of the client device 10. For example, when the user moves the client device 10 upward with respect to the surface of the virtual sphere, the immersive view controller 134 pans the immersive view upward, such as towards the sky or above a building. In other examples, when the user moves the client device 10 inward towards the surface of the virtual sphere to approach it, the immersive view controller 134 zooms in on the area depicted within the immersive view.

[0036] When the user selects a user control to view the 2D map display, the immersive view controller 134 can generate a virtual plane for the 2D map display, as will be described in more detail below with reference to FIG. 6. For example, the surface of the virtual plane can be positioned behind and parallel to the surface of the user interface 32 on the client device 10.

[0037] Next, when the user moves the client device 10 in a specific direction with respect to the virtual plane, the immersive view controller 134 detects the movement based on sensor data from the sensor 19 of the client device 10, such as an accelerometer or a gyroscope, and pans the 2D map in the movement direction of the client device 10. For example, when the user moves the client device 10 upward with respect to the surface of the virtual plane, if north is at the top of the digital map, the immersive view controller 134 pans the viewport northward. In other examples, when the user moves the client device 10 inward towards the surface of the virtual plane to approach it, the immersive view controller 134 zooms in on the area depicted within the viewport.

[0038] Note that although FIG. 1 shows the mapping application 122 as a stand-alone application, the functionality of the mapping application 122 can also be provided in the form of an online service accessible via a web browser executed on the client device 10, as a plug-in or extension of other software applications executed on the client device 10, etc. The mapping application 122 can generally be provided in different versions for different respective operating systems. For example, the manufacturer of the client device 10 can provide a software development kit (SDK) including the mapping application 122 for the Android (trademark) platform, another SDK for the iOS (trademark) platform, and so on.

[0039] In some embodiments, the server device 60 includes one or more processors 62 and one memory 64. The memory 64 can be a tangible non-transitory memory and can include any suitable memory module including random access memory (RAM), read-only memory (ROM), flash memory, other types of persistent memory, and the like. The memory 64 stores instructions executable on the processor 62 that constitute the immersive view generation engine 68, and this immersive view generation engine 68 can generate an immersive view of the 3D map from a virtual camera pose that matches the pose of the camera view (e.g., the rear camera view) within the client device 10.

[0040] In some embodiments, the immersive view generation engine 68 receives a request from the client device 10 for an immersive view of the 3D map. The request may include an indication of the pose of the rear camera view of the client device 10, or sensor data indicating the pose of the rear camera view of the client device 10, such as a real-world image from the rear camera, GPS data, accelerometer data, gyroscope data, compass data, magnetometer data, transceiver data indicating the signal strength of signals received from nearby wireless devices, and the like.

[0041] When the request includes sensor data indicating the pose of the rear camera view, the immersive view generation engine 68 determines the pose of the rear camera view based on the sensor data. For example, the immersive view generation engine 68 may determine the pose of the rear camera view by using a particle filter. For example, the immersive view generation engine 68 may determine a first pose with a first reliability from a GPS sensor, a second pose with a second reliability from an accelerometer, a third pose with a third reliability from a gyroscope, a fourth pose with a fourth reliability from a compass, a fifth pose with a fifth reliability from a transceiver (e.g., a position is determined based on the signal strength of a signal emitted by a nearby wireless beacon), and the like. The particle filter may combine the poses and reliabilities from each sensor to generate a pose with a lower margin of error than the reliability of each individual sensor. The particle filter may combine the poses and reliabilities in any suitable way, such as by assigning weights to each pose. The particle filter may also generate a probability distribution of the poses according to their respective reliabilities (e.g., using a Gaussian distribution where the reliability corresponds to two standard deviations). The particle filter may then use Bayesian estimation to combine the probability distributions of the poses to calculate a minimum mean square (MMS) estimate.

[0042] More specifically, the particle filter may obtain N random samples of a probability distribution called particles to represent the probability distribution, and assign a weight to each of the N random samples. Next, the particle filter combines the weighted particles to determine a pose having a lower error margin than the reliability of each sensor, with a reliability.

[0043] Furthermore, the immersive view generation engine 68 may determine the user's pose by receiving a rear camera view or image from the rear camera of the client device 10, and the rear camera view depicts a real-world image, for example, buildings, streets, vehicles, and people within the field of view of the rear camera. Next, the immersive view generation engine 68 may determine a pose corresponding to the rear camera view.

[0044] More specifically, the immersive view generation engine 68 may compare, for example, a rear camera view with some template camera views stored in the database 80. Each of the template camera views or images may be stored together with an indication of the camera viewpoint or position where the image was taken (e.g., GPS position specifying latitude and longitude coordinates, street address, etc.), an indication of the camera's orientation when the image was taken (e.g., indicating that the camera was facing north so that the camera view depicts the area north of the viewpoint), an indication of the scale or zoom level of the camera view, and / or an indication of the geographical area corresponding to the camera view including an indication of precise physical positions at various positions within the camera view based on the scale or zoom level. The orientation of the camera may include a three-axis orientation indicating the direction the camera is facing (e.g., east, west, north, south, etc.), the tilt angle of the camera (e.g., parallel to the ground), and whether the camera is in a horizontal or vertical position.

[0045] For example, in the case of a template camera view, the camera's perspective may be from the corner of the main street and State Street, and the camera was facing east. The template camera view may depict a width of 5 m, a length of 7 m, and a height of 3 m. Next, the immersive view generation engine 68 may create a mapping of the precise position of each pixel or group of pixels within the template camera view based on the camera's perspective, the camera's orientation, and the size of the template camera view. For example, if the template camera view is 5 m wide, the width of the image is 500 pixels, the orientation of the template camera view is facing east, and is perpendicular to the ground, each pixel may represent a physical width of approximately 1 cm. The camera perspective of the template camera view may be determined from the GPS position of the client device 10 that captured the camera view, and the orientation may be determined from the gyroscope and / or compass included in the client device 10 that captured the camera view.

[0046] In any case, the immersive view generation engine 68 may compare the rear camera view from the rear camera of the client device 10 with the template camera view and determine the camera's perspective and / or orientation of the rear camera view based on that comparison. For example, the immersive view generation engine 68 may use machine learning techniques such as random forest, boosting, nearest neighbor method, Bayesian network, neural network, support vector machine, etc. to compare the rear camera view from the rear camera of the client device 10 with the template camera view.

[0047] More specifically, the immersive view generation engine 68 can identify the visual features of each template camera view by detecting stable regions within the template camera view that are detectable regardless of blur, motion, distortion, orientation, lighting, scaling, and / or other changes within the camera pass. The stable regions can be extracted from the template camera view using SIFT (scale-invariant feature transform), SURF (speeded up robust features), FREAK (fast retina keypoint), BRISK (binary robust invariant scalable keypoints), or any other suitable computer vision technique. In some embodiments, the keypoints can be located in high-contrast regions of the template camera view, such as edges within the template camera view. A bounding box can be formed around the keypoint, and the portion of the template camera view created by the bounding box can be a feature.

[0048] The immersive view generation engine 68 can create a numerical representation of the features to generate a template feature vector, such as the width and height of the feature, the RGB pixel values of the feature, the pixel position of the center point of the feature within the image, and the like. Next, the immersive view generation engine 68 can use the feature vector as training data to create a machine learning model. For example, if the machine learning technique is the nearest neighbor method, the immersive view generation engine 68 can identify the visual features of the rear camera view from the rear camera of the client device 10 to generate a feature vector. Next, the immersive view generation engine 68 can compare the feature vector for the rear camera view from the rear camera of the client device 10 with the template feature vector to identify the template camera view having the template feature vector that is closest to the feature vector for the rear camera view from the rear camera of the client device 10.

[0049] Next, the immersive view generation engine 68 may determine the camera viewpoint and / or orientation of the rear camera view as the camera viewpoint and / or orientation of the identified template camera view. In still other embodiments, the immersive view generation engine 68 may determine the pose of the rear camera view based on any suitable combination of the sensor data and the comparison of the rear camera view to the template camera view. In other embodiments, the client device 10 may determine the camera viewpoint and / or orientation of the rear camera view using the techniques described above and provide an indication of the pose of the rear camera view to the server device 60.

[0050] In addition to determining the pose of the rear view camera, the immersive view generation engine 68 may also determine the distance between the user and the front camera, for example, by analyzing an image(s) from the front camera view of the front camera. For example, the immersive view generation engine 68 may receive sensor data indicative of the distance between the user and the front camera from the client device 10, such as a real-world image from the front camera. Next, the immersive view generation engine 68 may analyze the real-world image from the front camera to detect the user's face in the front camera view, for example, using face detection techniques.

[0051] Next, the immersive view generation engine 68 may determine the size of the user's face relative to the size of the front camera view. For example, the immersive view generation engine 68 may identify the length of the user's face in pixels compared to the length of the front camera view. Next, the immersive view generation engine 68 may determine the distance from the user to the client device 10 based on the size of the user's face relative to the size of the front camera view. For example, the size of the user's face relative to the size of the front camera view may be inversely proportional to the distance from the user to the client device 10. Next, the immersive view generation engine 68 may determine the initial zoom level of the immersive view based on the distance from the user to the client device 10. As the distance between the user and the client device 10 increases, the zoom level decreases. In other embodiments, the client device 10 may determine the distance from the user to the client device 10 using the above techniques and may provide an indication of the distance to the server device 60.

[0052] Next, the immersive view generation engine 68 may generate an immersive view having a pose of a virtual camera that matches the pose of the rear camera view and an initial zoom level corresponding to the identified zoom level. Next, the immersive view generation engine 68 may send the immersive view to the client device 10 for display. In some embodiments, the immersive view generation engine 68 may compare the generated immersive view with the rear camera view from the rear camera of the client device 10. The immersive view generation engine 68 may determine a difference metric between the generated immersive view and the rear camera view, for example, based on the pixel difference between the two views. Next, if the difference metric is less than a threshold difference, the immersive view generation engine 68 may determine that the generated immersive view is from the same pose as the rear camera view. In response to determining that the two views are from the same pose, the immersive view generation engine 68 may send the immersive view to the client device 10 for display. Otherwise, if the difference exceeds the threshold difference, the immersive view generation engine 68 may modify or change the immersive view. For example, the immersive view generation engine 68 may obtain a different immersive view from the database 80.

[0053] The immersive view generation engine 68 and the immersive view controller 134 can operate as components of a map display positioning system. Alternatively, the map display positioning system can include only server-side components and simply provide the immersive view to the immersive view controller 134 for presentation. In other words, the map display positioning technique in these embodiments can be implemented transparently to the immersive view controller 134. As another alternative, the overall functionality of the immersive view generation engine 68 can be implemented in the immersive view controller 134.

[0054] For simplicity, FIG. 1 shows server device 60 only as one instance of a server. However, server device 60 according to some embodiments includes one or more groups of server devices, each including one or more processors and capable of operating independently from other server devices. Server devices operating in such groups can process requests from client device 10 in a distributed manner where one operation associated with processing the request is executed on one server device and other operations associated with processing the same request are executed on other server devices, or according to any other suitable technique, individually (e.g., based on availability). For the purposes of this description, the term "server device" may refer to an individual server device or two or more groups of server devices.

[0055] Exemplary User Interface FIG. 2 shows an exemplary scenario where user 202 points the camera view of client device 10 at real-world image 200. Client device 10 may include a front camera 204 and a rear camera 206. Client device 10 includes sensors 19, such as an accelerometer and a gyroscope, to detect movement of client device 10 within the X, Y, and Z directions, where the X direction is the left-right direction with respect to user 202, the Y direction is the up-down direction with respect to user 202, and the Z direction is the direction of approaching user 202 and the direction of moving away from user 202. Immersive view controller 134 may generate a virtual sphere around user 202, for example, having a radius matching the distance d from user 202 to client device 10. Next, immersive view controller 134 may detect movement relative to the surface of the virtual sphere via sensors 19 and, in response, adjust the pose and / or zoom level of the virtual camera of the immersive view.

[0056] When user 202 points the camera view of rear camera 206 at real-world image 200, the user may select user control by mapping application 122 to view the immersive view of the 3D map having a virtual camera pose matching the camera pose of rear camera 206. In some embodiments, when user 202 activates rear camera 206 within mapping application 122, mapping application 122 automatically generates an immersive view.

[0057] In any case, in response to receiving the selection of user control or user 202 activating rear camera 206 within mapping application 122, immersive view controller 134 obtains sensor data from client device sensor 19, such as real-world images from rear camera 206, GPS data, accelerometer data, gyroscope data, compass data, magnetometer data, transceiver data indicating the signal strength of signals received from nearby wireless devices, etc. Next, immersive view controller 134 determines the pose of rear camera 206 by analyzing the sensor data using the techniques described above, or transmits the sensor data to server device 60 to determine the pose of rear camera 206.

[0058] Immersive view controller 134 also obtains sensor data to determine the distance d from user 202 to client device 10, such as real-world images from front camera 206. Next, immersive view controller 134 determines the distance d from user 202 to client device 10 by analyzing the sensor data using the above techniques, or transmits the sensor data to server device 60 to determine the distance d from user 202 to client device 10.

[0059] Next, the server device 60 generates an immersive view and provides the immersive view for display via the user interface 32 of the client device 10. FIG. 3 presents an exemplary immersive view 350 of a 3D map with interactive controls 352, and the immersive view 350 is from the same pose as the camera view of the real-world image 300 in front of the rear camera 206. As shown in FIG. 3, the immersive view 350 displays images of buildings and streets similar to the buildings and streets in the real world in front of the user. This enables the user to interact with the 3D map of the area in front of the user's eyes. For example, the user may select a user control such as an icon 352 that overlays a building, and this icon may provide additional information about the building when selected. The user may also pan or zoom the immersive view to view a distant location or view at a larger scale from the scale at which the user can see the real-world location. Further, if the user's view is blocked by a wall or building, the user may be able to "see through" the wall by viewing the immersive view 350 of the 3D map from the same pose as the view of the rear camera 206.

[0060] To pan or zoom the immersive view's 3D map at the pose and / or zoom level of another virtual camera, the user can move the client device 10 in the X, Y, or Z directions. FIG. 4 shows an exemplary immersive view 450 generated in response to the user moving the client device 10 upward in the Y direction. In response, the client device 10 adjusts the immersive view 450 to display images of buildings and other locations that are above the position shown in the immersive view 350. The immersive view 450 includes an image depicting the top of the Empire State Building with a label for the Empire State Building and user controls 452 for providing additional information about the Empire State Building. In some embodiments, when the user pans the 3D map, the immersive view controller 134 sends a request to the server device 60 for an immersive view at the panned virtual camera pose. In other embodiments, the immersive view controller 134 stores additional 3D map data for positions surrounding the area of the initial immersive view and uses the stored 3D map data to generate a new immersive view.

[0061] FIG. 5 shows an exemplary immersive view 550 generated in response to a user moving the client device 10 away from the user in the Z direction and inward with respect to the surface of the virtual sphere. In response, the client device 10 adjusts the 3D map to display the Empire State Building in the immersive view 550 at a larger scale than in the immersive view 450. In some embodiments, when the user zooms in or out on the 3D map, the immersive view controller 134 sends a request to the server device 60 for an immersive view at the adjusted zoom level. For example, the user may request to zoom the 3D map from a first zoom level to a second zoom level. Accordingly, the immersive view controller 134 sends a request to the server device 60 for an immersive view at the second zoom level. The server device 60 may generate a new immersive view of the 3D map at the second zoom level and provide the new immersive view to the immersive view controller 134 for display via the client device 10. In other embodiments, the immersive view controller 134 stores additional 3D map data for a plurality of zoom levels of the area of the initial immersive view and uses the stored 3D map data to generate a new immersive view.

[0062] By using an immersive view generated based on stored 3D map data, the visual quality of objects in the map, particularly distant objects, can be improved compared to a real-world map. This is because the quality of the rear camera 206 of the client device 10 does not need to be related to the quality of the immersive view, and thus a high-quality and detailed image can be consistently provided based on the 3D map data. Further, by using the stored 3D map data to generate a new immersive view at a new zoom level, the visual detail of the immersive view can be further improved compared to a real-world map where the zoom quality can be significantly limited by the quality of the rear camera 206 and / or other imaging devices, components or processes of the client device 10. As a result, the immersive view provides a consistently improved detailed map view regardless of the zoom level and the quality of the camera.

[0063] In some embodiments, as the user moves the client device 10, the user may select a user control to lock the virtual camera pose of the immersive view so that the immersive view does not change. The user control may be, for example, a long-press gesture such that the immersive view does not change as long as the user holds a finger on the user interface 32 of the client device 10. In other embodiments, the immersive view may not change by default, and the user may need to perform a long-press gesture while moving the client device 10 to pan or zoom the client device 10. By locking the virtual camera pose of the immersive view, the user can, for example, pan to the right within the range that the user can move the arm to the right. Then while the user moves the arm back towards the user's body, the user can lock the virtual camera pose. Next, the user can unlock the virtual camera pose, for example, by releasing the long-press gesture. And by moving the arm to the right again, the user can continue to pan further to the right.

[0064] As described above, in some embodiments, the mapping application 122 includes user controls for viewing a 2D map of the area presented within the immersive view. For example, the user may perform a specific touch gesture on the user interface 32 to transition from the immersive view to the 2D map. In other embodiments, the mapping application 122 may present a 2D map that by default has a specific viewport.

[0065] FIG. 6 shows a 2D map 600 having a specific viewport 650 presented via the user interface 32 of the client device 10. As shown in FIG. 6, the specific viewport 650 is a visible portion of the larger 2D map 600. In some embodiments, the immersive view controller 134 generates a virtual plane that may be positioned behind and parallel to the surface of the user interface 32 on the client device 10. When the user then moves the client device in a specific direction with respect to the virtual plane, the immersive view controller 134 adjusts the viewport accordingly.

[0066] For example, FIG. 7 shows an exemplary scenario where the user moves the client device 10 to the right in the X direction. Accordingly, when the viewport 750 is positioned such that north is above the viewport 750, the immersive view controller 134 adjusts the viewport 750 to present a portion of the 2D map 700 that is east of the viewport 650. FIG. 8 shows an exemplary scenario where the user rotates the client device 10 counterclockwise. In response, the immersive view controller 134 rotates the viewport 850 to present a portion of the 2D map 800 where the top of the viewport 850 faces northwest rather than north.

[0067] In another example, FIG. 9 shows an exemplary scenario where the user moves the client device 10 inward and closer to the surface of the virtual plane in the Z direction. Accordingly, the immersive view controller 134 zooms in on the 2D map 900 and presents a viewport 950 that is zoomed in compared to the viewport 650. FIG. 10 shows an exemplary scenario where the user moves the client device 10 outward and away from the surface of the virtual plane in the Z direction. Accordingly, the immersive view controller 134 zooms out on the 2D map 1000 and presents a viewport 1050 that is zoomed out compared to the viewport 650.

[0068] In yet another example, FIG. 11 shows an exemplary scenario where the user tilts the client device 10 towards the user while locking the position of the virtual plane (e.g., via user controls). The user controls may be the same as or different from the user controls for locking the position of the viewport. Since the position of the virtual plane is locked, the position of the surface of the virtual plane does not change. Accordingly, when the user tilts the client device 10 and the position of the surface of the virtual plane does not change, the surface of the virtual plane is tilted with respect to the client device 10. As a result, the immersive view controller 134 presents a tilted view of the viewport 1150, so that map features closer to the top of the viewport 1150 appear further away than map features closer to the bottom of the viewport 1150. In other embodiments, the user does not need to lock the position of the viewport 1150 (e.g., via user controls) to tilt the viewport 1150 with respect to the surface of the virtual plane. Instead, the immersive view controller 134 locks the position of the virtual plane by default and tilts the viewport 1150 in response to the user tilting the client device 10. In yet other embodiments, the surface of the virtual plane may be parallel to the ground regardless of the tilt angle of the client device 10. In these embodiments, the viewport does not tilt.

[0069] Exemplary method for localizing a user within a map display FIG. 12 shows a flowchart of an exemplary method 1200 for localizing a user within a digital map using an immersive view of the user's location. The method can be implemented in a set of instructions that can be stored in a computer-readable memory and executed by one or more processors of the client device 10. For example, the method can be implemented by an immersive view controller 134 within a mapping application 122.

[0070] At block 1202, the immersive view controller 134 determines the location of the client device 10. The immersive view controller 134 may also determine the orientation of the client device 10. More specifically, the immersive view controller 134 may determine the position and orientation (the “rear camera pose”) of the rear camera of the client device 10. The immersive view controller 134 may determine the location and orientation by obtaining sensor data from sensors 19 within the client device 10, such as real-world images from the rear camera, GPS data, accelerometer data, gyroscope data, compass data, magnetometer data, transceiver data indicating the signal strength of signals received from nearby wireless devices, etc. The immersive view controller 134 may then analyze the sensor data to determine the rear camera pose, for example, by comparing a real-world image from the rear camera with template camera views stored in the database 80, and / or by analyzing GPS data, accelerometer data, gyroscope data, compass data, magnetometer data, transceiver data, etc.

[0071] In other embodiments, the immersive view controller 134 may send the sensor data to the server device 60, and the server device may analyze the sensor data to determine the rear camera pose. The server device 60 may then provide an indication of the rear camera pose to the client device 10.

[0072] In block 1206, the immersive view controller 134 presents an immersive view of the 3D map from the perspective of a virtual camera having an orientation and position that match the orientation and position of the rear camera view within the client device 10. The immersive view controller 134 may send a request for an immersive view of the 3D map to the server device 60. The request may include an indication of the pose of the rear camera view of the client device 10, or sensor data indicating the pose of the rear camera view of the client device 10, such as a real-world image from the rear camera, GPS data, accelerometer data, gyroscope data, compass data, magnetometer data, transceiver data indicating the signal strength of signals received from nearby wireless devices, and the like.

[0073] The request may also include an indication of the distance between the user and the front camera view of the client device 10, or sensor data indicating the distance between the user and the front camera view of the client device 10, such as a real-world image from the front camera. Next, the server device 60 may analyze the real-world image from the front camera to identify the user within the front camera view, for example, using face detection techniques, and determine the distance from the user to the client device 10 based on, for example, the size of the user's face within the front camera view.

[0074] Next, the server device 60 may determine the pose of the virtual camera for the immersive view based on the pose of the rear camera view of the client device 10. The server device 60 may also determine the initial zoom level for the immersive view based on the distance from the user to the client device 10. Next, the server device 60 may obtain an immersive view having a matching virtual camera pose and zoom level from the database 80 and provide the immersive view to the immersive view controller 134 for display.

[0075] FIG. 13 shows a flowchart of an exemplary method 1300 for generating an immersive view at a user's location to localize the user within a digital map. The method can be implemented in a set of instructions that can be stored in a computer-readable memory and executed by one or more processors of the server device 60. For example, the method can be implemented by the immersive view generation engine 68.

[0076] In block 1302, the immersive view generation engine 68 obtains sensor data from the client device 10. The sensor data can include, for example, a real-world image from a rear camera, GPS data, accelerometer data, gyroscope data, compass data, magnetometer data, transceiver data indicating the signal strength of signals received from nearby wireless devices, etc. The sensor data may be obtained as part of a request for an immersive view of the 3D map from the client device 10.

[0077] Next, in block 1304, the immersive view generation engine 68 determines the position and orientation of the client device 10 based on the sensor data. For example, the immersive view generation engine 68 may compare a real-world image from a rear camera with a template camera view stored in the database 80. Additionally or alternatively, the immersive view generation engine 68 may analyze GPS data, accelerometer data, gyroscope data, compass data, magnetometer data, transceiver data, etc. to determine the position and orientation of the client device 10.

[0078] In block 1306, the immersive view generation engine 68 generates an immersive view of the 3D map from the parse of a virtual camera having an orientation and position that match the orientation and position of the rear camera view within the client device 10. For example, the immersive view generation engine 68 may generate an immersive view by obtaining 3D map data from a database 80 having a virtual camera pose that matches the rear camera pose of the client device 10. Next, in block 1308, the immersive view generation engine 68 provides the immersive view to the client device 10 for display. In some embodiments, the immersive view generation engine 68 generates multiple immersive views for different zoom levels or virtual camera poses within the vicinity of the rear camera pose of the client device 10. Since the immersive view generation engine 68 provides multiple immersive views to the client device 10, the user can pan or zoom the 3D map without the client device 10 making additional requests to the server device 60 for immersive views in the same general area.

[0079] Other Considerations The following additional considerations apply to the foregoing discussion. Throughout this specification, multiple examples may implement components, operations, or structures described as a single example. Individual operations of one or more methods are illustrated and described as separate operations, but one or more of the individual operations may be performed together and nothing requires the operations to be performed in the order illustrated. Structures and functionality represented as separate components in example configurations may be implemented as a combined structure or component. Similarly, structures and functionality represented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements are within the scope of the subject matter described herein.

[0080] Certain embodiments are described herein as including logic or several components, modules, or mechanisms. A module can be either a software module (e.g., the code is embodied on a machine-readable medium or in a transmitted signal) or a hardware module. A hardware module is a tangible unit capable of performing certain operations and can be configured or arranged in a certain manner. In an example of an embodiment, one or more computer systems (e.g., stand-alone, client, or server computer systems) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or a portion of an application) as a hardware module that operates to perform certain operations as described herein.

[0081] Unless otherwise specified, discussions herein using terms such as "processing," "computing," "calculating," "determining," "representing," "displaying," and the like can refer to actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more of a memory (e.g., volatile memory, non-volatile memory, or a combination thereof), a register, or other machine components that receive, store, transmit, or display information.

[0082] As used herein, any reference to "one embodiment" or "an embodiment" means that the particular element, function, structure, or characteristic described in relation to that embodiment is included in at least one embodiment. The phrase "in one embodiment" appearing in various places in this specification does not necessarily refer to the same embodiment.

[0083] Some embodiments may be described using the terms "coupled" and "connected" and derivatives thereof. For example, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other but cooperate or interact with each other. The present embodiment is not limited to this context.

[0084] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," or any other variation thereof are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" refers to an inclusive or and not an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0085] In addition, the use of "a" or "an" is employed to describe elements and components of the embodiments described herein. This is done merely for convenience and to give a general sense of the various embodiments. This description should be read to include one or at least one and, unless clearly indicated to the contrary, the singular also includes the plural.

[0086] Upon reading this disclosure, those skilled in the art will understand yet another alternative structural design and functional design for positioning a user within a map display through the principles disclosed herein. Thus, while specific embodiments and applications are shown and described, it is to be understood that the disclosed embodiments are not limited to the exact construction and components disclosed herein. Various modifications, changes and variations that are obvious to those skilled in the art may be made in the arrangement, operation and details of the methods and apparatuses disclosed herein without departing from the spirit and scope of the appended claims.

Claims

1. A method for localizing a user within a digital map using an immersive view of the user's location, comprising: determining, by one or more processors in a client device, a location of the client device; determining, by the one or more processors, an orientation of the client device; presenting, by the one or more processors, an immersive view of a three-dimensional (3D) map from a parse of a virtual camera having an orientation and position matching the orientation and position of the camera view within the client device; A method comprising the above.

2. The method according to claim 1, wherein determining the orientation of the client device includes determining the orientation of a rear camera view of the client device.

3. The method according to claim 1 or 2, further comprising determining, by the one or more processors, an initial zoom level for presenting the immersive view based on a distance between the user and the client device.

4. The method according to claim 3, further comprising determining, by the one or more processors, the distance between the user and the client device based on a size of the user's face within a front camera view of the client device.

5. Determining the location and orientation of the client device includes: receiving, by the one or more processors, sensor data from one or more sensors in the client device, the one or more sensors including at least one of a positioning sensor, a compass, a gyroscope, an accelerometer, a transceiver, or a camera; determining, by the one or more processors, the location and orientation of the client device using the sensor data; The method according to any one of claims 1 to 4, comprising the above.

6. Determining the position and orientation of the client device using the sensor data includes the one or more processors providing a real-world image from the camera view of the camera to a server device, comparing the real-world image with a plurality of template real-world images from a plurality of positions and orientations stored in a real-world image database, identifying at least one of the plurality of template real-world images that matches the real-world image from the camera, and identifying the position and orientation of the client device based on the position and orientation of the identified template real-world image. The method according to claim 5.

7. detecting, by the one or more processors, that the client device has moved in a particular direction; repositioning, by the one or more processors, the immersive view based on the direction of movement of the client device; The method according to any one of claims 1 to 6, further comprising.

8. Repositioning the immersive view based on the direction of movement of the client device includes identifying the surface of a virtual sphere surrounding the user; panning the immersive view within the direction of movement of the client device relative to the virtual sphere; The method according to claim 7, comprising.

9. Repositioning the immersive view based on the direction of movement of the client device includes identifying the surface of a virtual sphere surrounding the user; adjusting the zoom level of the viewport in response to detecting that the client device has approached or moved away from the surface of the virtual sphere; The method according to claim 7, comprising.

10. receiving user input requesting to lock the position of the virtual camera for the immersive view to the current position; detecting, by the one or more processors, that the client device has moved in a particular direction; presenting the immersive view from the perspective of the virtual camera at the current position without repositioning the immersive view based on the direction of movement of the client device by the one or more processors; The method according to any one of claims 1 to 9, further comprising

11. The method according to claim 10, wherein the user input is a long-press gesture on a user interface of the client device, and the immersive view does not change its position while the user's finger remains in contact with the user interface.

12. A client device for localizing a user within a digital map using an immersive view of the user's location, comprising a user interface, one or more processors, and a computer-readable memory having computer-executable instructions that, when executed by the one or more processors, cause the client device to determine the position of the client device, determine the orientation of the client device, and present an immersive view of a three-dimensional (3D) map from a perspective of a virtual camera having an orientation and position that match the orientation and position of a camera view within the client device via the user interface. a computer-readable memory, and a client device comprising.

13. The client device according to claim 12, wherein the orientation of the client device is determined based on the orientation of a rear camera view of the client device.

14. The client device according to claim 12 or 13, wherein the instructions further cause the client device to determine an initial zoom level for presenting the immersive view based on a distance between the user and the client device.

15. The client device according to claim 14, wherein the instructions further cause the client device to determine the distance between the user and the client device based on a size of the user's face within a front camera view of the client device.

16. To determine the position and orientation of the client device, the instructions cause the client device to receive sensor data from one or more sensors including at least one of a positioning sensor, a compass, a gyroscope, an accelerometer, a transceiver, or a camera. Determining the position and orientation of the client device using the sensor data; The client device according to any one of claims 12 to 15, which causes the above.

17. A method for generating an immersive view at the position of a user to localize the user within a digital map, comprising: obtaining sensor data from one or more sensors in a client device by one or more processors; determining, by the one or more processors, the orientation and position of the client device based on the sensor data; generating, by the one or more processors, an immersive view of a three-dimensional (3D) map from a parse of a virtual camera having an orientation and position matching the orientation and position of the camera view in the client device; providing, by the one or more processors, the immersive view for display via the client device; A method comprising the above.

18. The method according to claim 17, wherein the one or more sensors include at least one of a positioning sensor, a compass, a gyroscope, an accelerometer, a transceiver, or a camera.

19. The method according to claim 17 or 18, wherein determining the orientation of the client device includes determining the orientation of the rear camera view of the client device.

20. The method according to any one of claims 17 to 19, further comprising determining, by the one or more processors, an initial zoom level of the immersive view based on a distance between a user and the client device.

21. receiving, by the one or more processors, a request to change a zoom level of the 3D map from a first zoom level to a second zoom level; generating, by the one or more processors, a new immersive view of the 3D map at the second zoom level; providing, by the one or more processors, the new immersive view for display via the client device; The method according to any one of claims 17 to 19, further comprising the above.

Images