Point-based orientation

Through the image sensor detecting multiple lines and determining the direction of vanishing, the problem of unstable measurement coordinate system of the inertial measurement unit is solved, and the stable display of virtual content in a mobile environment is achieved.

CN119006579APending Publication Date: 2024-11-22APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410618705.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-18
Filing Date
2024-05-17
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In existing extended reality (XR) environments, coordinate systems based on inertial measurement units (IMU) measurements may be unstable in user perception, especially in moving or tilting environments, making it difficult to accurately define the downward direction.

Method used

By capturing images of the physical environment using image sensors, detecting multiple lines, and determining the direction of disappearance based on the Random Sample Consistency (RANSAC) algorithm and voting algorithm, defining a more stable downward direction.

Benefits of technology

It realizes that the coordinate system perceived by the user is more closely matched in a mobile or tilted environment, ensuring the stable display of virtual content, especially in mobile environments such as vehicles and aircraft, where the position of virtual objects is more accurate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006579A_ABST
    Figure CN119006579A_ABST
Patent Text Reader

Abstract

The invention relates to vanishing point-based orientation. In one embodiment, a method of determining a downward direction is performed by a device including one or more processors, a non-transitory memory, and an image sensor. The method includes capturing an image of a physical environment using the image sensor. The method includes detecting a plurality of lines within the image of the physical environment. The method includes determining one or more disappearance directions based on the plurality of lines. The method includes defining a downward direction based on the one or more disappearing directions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to systems, methods, and apparatus for defining a coordinate system. Background Art

[0002] In various implementations, an extended reality (XR) environment is presented by a head mounted device (HMD). Various HMDs include a scene camera that captures images of a physical environment (e.g., a scene) in which a user is located and a display that overlays virtual content on the physical environment. In various implementations, the virtual content is displayed according to a coordinate system, e.g., a virtual ball floating upward or a virtual ball falling downward. In various implementations, determining the coordinate system (e.g., which direction on the display corresponds to "up" or "down") is based on a gravity vector measured by an inertial measurement unit. However, in various cases, such a coordinate system may be unstable or skewed relative to the user's perception. Summary of the invention

[0003] Various embodiments disclosed herein include devices, systems, and methods for determining a downward direction. In various embodiments, the method is performed by a device including one or more processors, a non-volatile memory, and an image sensor. The method includes capturing an image of a physical environment using the image sensor. The method includes detecting a plurality of lines within the image of the physical environment. The method includes determining one or more vanishing directions based on the plurality of lines. The method includes defining a downward direction based on the one or more vanishing directions.

[0004] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors. One or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and a device for executing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0006] Figure 1A-Figure 1C is a block diagram of an exemplary operating environment according to some implementations.

[0007] Figure 2 An image of a physical environment is shown.

[0008] Figure 3 A plurality of lines detected in an image of a physical environment are shown.

[0009] Figure 4 A vanishing point based on multiple lines is shown.

[0010] Figure 5 Virtual content is shown displayed over an image of a physical environment.

[0011] Figure 6 is a flow chart of a method for determining a downward direction based on one or more vanishing points.

[0012] Figure 7 is a block diagram of an exemplary electronic device according to some implementations.

[0013] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0014] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0015] As described above, in various implementations, virtual content is displayed on the physical environment (either directly using a transparent perspective display or using an opaque perspective display to display on an image of the physical environment). In various implementations, a coordinate system of the physical environment is defined to include at least a definition of a downward direction. In various implementations, virtual content is displayed according to the coordinate system. For example, in various implementations, a virtual application window is displayed to have an edge perpendicular to the downward direction. In various implementations, a virtual ball is displayed to fall in the downward direction. In various implementations, a virtual ball is displayed to float upward opposite to the downward direction.

[0016] In various implementations, determining the downward direction is based on a vector measured by an inertial measurement unit (IMU). When stationary, the vector is generally consistent with the direction of gravity (e.g., downward). However, in various situations, the vector may be unstable or skewed relative to the user's perception. For example, when tilted, the vector points in the direction of gravity instead of pointing to the ground. As another example, in a moving vehicle, when the vehicle accelerates or decelerates, the vector changes due to the acceleration and does not consistently point in the direction of gravity or the vehicle floor.

[0017] Therefore, in various implementations, a coordinate system is determined based on the vanishing point of the image of the physical environment. Such a coordinate system may more closely match the user's perception (e.g., pointing "downward" to the ground or floor) and may be more stable, especially in a mobile environment.

[0018] Figure 1A A car 110 is shown on a ramp 111. The IMU within the car 110 measures a first vector 115 in the direction of gravity. However, using the vanishing point of the image from within the car 110, a second vector 117 pointing to the floor of the car 110 and perpendicular to the ramp 111 is determined.

[0019] Figure 1B An aircraft 120 is shown performing a banking maneuver. An IMU within the aircraft 120 measures a first vector 125 in the direction of gravity. However, using the vanishing point of the image from within the aircraft 120, a second vector 127 pointing toward the floor of the aircraft 120 is determined.

[0020] Figure 1C Train 130 is shown accelerating according to acceleration vector 131. The IMU within train 130 measures a first vector 135 that does not point in the direction of gravity, but is affected by both gravity and the acceleration of train 130. However, using the vanishing point of the image from within train 130, a second vector 137 is determined that points toward the floor of train 130 and is in the direction of gravity.

[0021] Figure 2 An image of a physical environment 200 is shown. In various implementations, the physical environment is a moving vehicle. In various implementations, the physical environment is the interior of a subway vehicle. The physical environment includes a plurality of seats 210A-210F, an aisle 212 separating the plurality of groups of seats 210A-210F, a door 214 at the end of the aisle 214, a window 216, and a ceiling 218 with lights 220A-220B on both sides.

[0022] Figure 3 An image of the environment 200 is shown overlaid with a plurality of lines 310A- 310H, 320A- 320H detected in the image of the environment 200 .

[0023] Figure 4 A plurality of lines 310A- 310H, 320A- 320H are shown having a first vanishing point 410 and a second vanishing point 420 determined based on the plurality of lines 310A- 310H, 320A- 320H.

[0024] In various implementations, the first vanishing point 410 and the second vanishing point 420 are determined using one or more random sample consensus (RANSAC) algorithms. For example, in various implementations, the first vanishing point 410 is determined by a 2-line RANSAC algorithm, where for n iterations, two of the plurality of lines 310A-310H, 320A-320H are selected and intersection points are determined. Each of the two of the plurality of lines 310A-310H, 320A-320H may be selected from a first group of the plurality of lines 310A-310H or a second group of the plurality of lines 320A-320H. Thus, n intersection points are determined as candidate vanishing points. Each candidate vanishing point has a number of lines of the plurality of lines as inliers, where each line in the inliers (when extended in two directions) passes within a certain threshold distance of the candidate vanishing point. The candidate vanishing point with the largest inliers is selected as a preliminary first vanishing point. The preliminary first vanishing point is then refined based on the corresponding inliers. For example, in various implementations, the first vanishing point 410 is the point that is closest to each line in the inlier set of lines (when extended in both directions) to the preliminary first vanishing point using the minimum mean square error measurement. Figure 4 , the interior point group includes a first group of the plurality of lines 310A- 310H used to determine the first vanishing point 410 .

[0025] The first vanishing point 410 defines a first vector 440A between the principal point 400 of the perspective of the image of the physical environment 200 and the first vanishing point 410. Furthermore, the first vector defines a first direction.

[0026] In various implementations, a second vanishing point 420 is also determined using a 2-line RANSAC algorithm on the first set of multiple lines 320A-320H excluding multiple lines 310A-310H. The second vanishing point 420 defines a second vector 440B between the principal point 400 of the perspective of the image of the physical environment 200 and the second vanishing point 420. In addition, the second vector defines a second direction.

[0027] In various implementations, the second direction is determined by a voting algorithm. For example, in various implementations, the image of the physical environment 200 is transformed so that the first vanishing point 410 is aligned with the principal point of view of the image of the physical environment 200. Voting is applied to each line direction between 0 and π, and the mean shift is used to find the second direction (and the corresponding second vector). In various implementations, the third vector 440C defining the third direction is determined to be proportional to the cross product of the first vector 440A and the second vector 440B.

[0028] In various specific implementations, the first vanishing point 410 and the second vanishing point 420 are determined using a 3-line RANSAC algorithm, wherein three of 310A–310H, 320A–320H are selected for n iterations. Each of the three lines of the plurality of lines 310A–310H, 320A–320H may be selected from the first group of the plurality of lines 310A–310H or the second group of the plurality of lines 320A–320H. The candidate first vanishing point is determined as the intersection of the first line of the three lines and the second line. The candidate first vanishing point defines a first candidate direction. In addition, the third line of the three lines defines a candidate second direction perpendicular to the candidate first direction. In addition, the candidate third vanishing direction is determined using the cross product of the vectors along the candidate first direction and the candidate second direction. Therefore, n candidate coordinate systems are generated, each candidate coordinate system being defined by a corresponding candidate first direction, a candidate second direction, and a candidate third direction. Each candidate coordinate system has several lines of the plurality of lines as an inner point group. The candidate coordinate system having the largest interior point group is selected as a coordinate system including the first direction, the second direction, and the third direction.

[0029] In various implementations, the third direction is assumed to be a downward direction. In various implementations, the downward direction is selected from the first direction, the second direction, and the third direction as the direction closest to the vector output from the IMU. It is worth noting that the third direction is neither the direction of the vector provided by the IMU nor the vertical direction in the image of the physical environment.

[0030] Figure 5 An image of a physical environment 200 with virtual content overlaid thereon is shown. The virtual content includes a virtual web browser window 510 and a virtual ball 520. The virtual web browser window 510 is aligned with a third direction. In particular, the left and right edges of the virtual web browser window 510 are parallel to the third direction, and the top and bottom edges of the virtual web browser window 510 are perpendicular to the third direction. The virtual ball 520 bounces off the aisle 212, moving up and down along the third direction.

[0031] Figure 6is a flowchart representation of a method for determining a downward direction according to some specific implementations. In various specific implementations, method 600 is performed by a device having one or more processors, non-transitory memory, and an image sensor. In some specific implementations, method 600 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, method 600 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0032] Method 600 begins in block 610, where a device captures an image of a physical environment using an image sensor. In various implementations, the physical environment is a moving vehicle. For example, in various implementations, the physical environment is a car, a train, an airplane, or a subway.

[0033] The method 600 continues in block 620 where the device detects a plurality of lines within the image of the physical environment. Various line detection algorithms may be used, including Hough transforms or convolution-based techniques.

[0034] Method 600 continues in block 630, where the device determines one or more vanishing directions based on the plurality of lines. In various implementations, determining the one or more vanishing directions includes determining a first vanishing point. In various implementations, determining the first vanishing point is based on a random sample consensus (RANSAC) algorithm. In various implementations, the RANSAC algorithm defines a first subset of the plurality of lines. In various implementations, determining the one or more vanishing directions includes determining a second vanishing direction. In various implementations, determining the second vanishing direction includes determining a second vanishing point based on the plurality of lines excluding the first subset of the plurality of lines. In various implementations, determining the second vanishing point is based on the RANSAC algorithm. In various implementations, determining the second direction is based on a voting algorithm. In various implementations, determining the one or more vanishing directions includes determining a third vanishing direction based on the plurality of lines. In various implementations, determining the third vanishing direction is based on a cross product of the first direction and the second direction. In various implementations, determining the one or more vanishing directions includes determining the first vanishing direction, the second vanishing direction, and the third vanishing direction based on a 3-line RANSAC algorithm.

[0035] The method 600 continues in block 640, where the device defines a downward direction based on the one or more disappearing directions. In various implementations, the method 600 includes selecting a third direction as the downward direction. In various implementations, the method 600 includes selecting the first direction, the second direction, or the third direction as the downward direction based on a vector from an inertial measurement unit.

[0036] In various implementations, method 600 includes displaying virtual content based on a downward direction. In various implementations, displaying virtual content based on a downward direction includes displaying a virtual object aligned with the downward direction. For example, in Figure 5 In the embodiment, the virtual web browser window 510 is aligned with the downward direction. In various embodiments, displaying virtual content based on the downward direction includes displaying a virtual object moving in the downward direction. Figure 5 , the virtual ball 520 bounces up and down in the downward direction.

[0037] In various implementations, determining a downward direction based on a detected disappearing direction is more robust, particularly in a moving physical environment. Thus, in various implementations, the downward direction is repeatedly determined using the disappearing direction (rather than, for example, directly from an inertial measurement unit). Thus, in various implementations, method 600 includes capturing a second image of the physical environment using an image sensor; detecting a plurality of lines within the second image; determining one or more updated disappearing directions based on the plurality of lines within the second image; and updating the downward direction based on the updated one or more disappearing directions. In various implementations, method 600 also includes updating the display of virtual content based on the updated downward direction.

[0038] Figure 7 7 is a block diagram of an example of an electronic device 700 according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the electronic device 700 includes one or more processing units 702 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 706, one or more communication interfaces 708 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 710, one or more XR displays 712, one or more optional internal-facing and / or external-facing image sensors 714, memory 720, and one or more communication buses 704 for interconnecting these and various other components.

[0039] In some implementations, the one or more communication buses 704 include circuits for interconnecting system components and controlling communications between system components. In some implementations, the one or more I / O devices and sensors 706 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0040] In some implementations, one or more XR displays 712 are configured to provide an XR experience to the user. In some implementations, one or more XR displays 712 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some implementations, one or more XR displays 712 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the electronic device 720 includes a single XR display. As another example, the electronic device includes an XR display for each eye of the user. In some implementations, one or more XR displays 712 can present MR and VR content.

[0041] In some implementations, the one or more image sensors 714 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (either sensor may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors 714 are configured to face forward so as to acquire image data corresponding to the physical environment that the user would see when the electronic device 700 is not present (and thus may be referred to as a scene camera). The one or more optional image sensors 714 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and the like.

[0042] The memory 720 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some specific implementations, the memory 720 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 720 optionally includes one or more storage devices remotely located from the one or more processing units 702. The memory 720 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 720 or the non-transitory computer-readable storage medium of the memory 720 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 730 and an XR rendering module 740.

[0043] The operating system 730 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the XR rendering module 740 is configured to present XR content to a user via one or more XR displays 712. To this end, in various implementations, the XR rendering module 740 includes a data acquisition unit 742, an orientation unit 744, an XR rendering unit 746, and a data transmission unit 748.

[0044] In some implementations, the data acquisition unit 742 is configured to obtain data (e.g., presentation data, interaction data, sensor data, location data, etc.) from other components of the electronic device 700 and / or other electronic devices. To this end, in various implementations, the data acquisition unit 742 includes instructions and / or logic components for instructions, as well as heuristics and metadata for heuristics.

[0045] In some implementations, the orientation unit 744 is configured to determine the downward direction based on one or more disappearing directions of the image. To this end, in various implementations, the orientation unit 744 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0046] In some implementations, the XR rendering unit 746 is configured to display virtual content based on a downward direction via the one or more XR displays 712. To this end, in various implementations, the XR rendering unit 746 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0047] In some implementations, the data transmission unit 748 is configured to transmit data (e.g., presentation data, location data, etc.) to other components of the electronic device 700 and / or other electronic devices. To this end, in various implementations, the data transmission unit 748 includes instructions and / or logic components for instructions as well as heuristics and metadata for the heuristics.

[0048] Although the data acquisition unit 742, orientation unit 744, XR rendering unit 746, and data transmission unit 748 are shown as residing on a single device (e.g., electronic device 700), it should be understood that in other specific implementations, any combination of the data acquisition unit 742, orientation unit 744, XR rendering unit 746, and data transmission unit 748 may be located in separate computing devices.

[0049] also, Figure 7 More as a functional description of various features that may be present in a particular embodiment, as opposed to a schematic diagram of the structure of the specific implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 7 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific implementation.

[0050] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0051] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0052] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0053] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method comprising: At a device including one or more processors, non-transitory memory, and an image sensor: capturing an image of a physical environment using the image sensor; detecting a plurality of lines within the image of the physical environment; determining one or more disappearing directions based on the plurality of lines; and A downward direction is defined based on the one or more disappearing directions. 2 . The method of claim 1 , wherein determining the one or more vanishing directions comprises determining a first vanishing point and determining a first vanishing direction based on the first vanishing point. 3 . The method of claim 2 , wherein determining the first vanishing point is based on a random sample consensus (RANSAC) algorithm that defines a first subset of the plurality of lines. 4 . The method of claim 3 , wherein determining the one or more vanishing directions comprises determining a second vanishing direction based on a second vanishing point, the second vanishing point being determined based on a RANSAC algorithm that excludes the first subset of the plurality of lines. The method according to claim 4 , wherein determining the one or more disappearing directions comprises determining a second disappearing direction based on a voting algorithm. The method of claim 4 , wherein determining the one or more vanishing directions comprises determining a third direction based on a cross product of the first vanishing direction and the second vanishing direction. 7 . The method according to claim 1 , wherein determining the one or more vanishing directions comprises determining a first vanishing direction, a second vanishing direction, and a third vanishing direction based on a trilinear RANSAC algorithm. The method of claim 6 , further comprising selecting the third direction as the downward direction.

9. The method of claim 6, further comprising selecting the first direction, the second direction, or the third direction as the downward direction based on a vector from an inertial measurement unit.

10. The method of claim 1, further comprising displaying virtual content based on the downward direction. The method of claim 10 , wherein displaying the virtual content based on the downward direction comprises displaying a virtual object aligned with the downward direction. 12 . The method of claim 10 , wherein displaying the virtual content based on the downward direction comprises displaying a virtual object moving along the downward direction.

13. The method according to claim 1, further comprising: capturing a second image of the physical environment using the image sensor; detecting a plurality of lines within the second image; determining updated one or more disappearance directions based on the plurality of lines within the second image; as well as The downward direction is updated based on the updated one or more disappearing directions. The method of claim 13 , further comprising updating a display of virtual content based on the updated downward direction.

15. A device comprising: Image sensor; one or more processors; non-transitory memory; as well as One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to perform the method according to any one of claims 1 to 14.

16. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device, cause the device to perform the method according to any one of claims 1 to 14.