Method for simultaneous localization and mapping

By identifying and converting the set of points of objects in the second time window in SLAM, the problems of large delay and low efficiency in the prior art are solved, and low latency and efficient object positioning and mapping are achieved, and user experience is improved.

CN119096271APending Publication Date: 2024-12-06VOXELSENSORS SRL
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380036320.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-02
Filing Date
2023-04-25
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing SLAM methods have high latency, low efficiency, and may lead to motion sickness and incomplete scanning when processing fast moving objects. They cannot quickly and accurately update the location and orientation of the objects, resulting in poor user experience.

Method used

By identifying the set of points of objects within the second time window and converting them into a more dense set of 3D positions within the first time window, the transformation is determined for best fit, reducing the number of identified points, and achieving low latency and efficient SLAM.

Benefits of technology

It realizes low latency, accurate and efficient object positioning and mapping, reduces power consumption, prevents motion sickness, provides a smooth user experience, and only partially identifying object points can achieve accurate results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119096271A_ABST
    Figure CN119096271A_ABST
Patent Text Reader

Abstract

The invention relates to a method for sensing. The method comprises the step of identifying a second set of locations (2) of a point of at least one object (3) in the scene (4) relative to a first reference point (28), such as preferably a first viewpoint, within a second time window (T2). The method further comprises a step of converting the second set of locations (2) into a second set of 3D positions (6). The method further comprises a step of determining a transformation to the second set of 3D positions (6) such that the transformed second set of 3D positions (6) best fits a first set of 3D positions (5), where the first set of 3D positions (5) is denser than the second set of 3D positions (6), and where the first set of 3D positions (5) is within a first time window (T1) preceding the second time window (T2), and where the first set of 3D positions (5) is within a second time window (T2) preceding the second time window (T2). And wherein the second time window (T2) is shorter, preferably at least twice, than the first time window (T1).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for simultaneous localization and mapping (SLAM). In particular, the present invention relates to a low-latency method for SLAM. Background Art

[0002] The art describes different methods for acquiring 3D information of an environment and objects in said environment, for the purpose of, for example, creating an augmented reality of said environment. Simultaneous Localization and Mapping (SLAM) allows building a map of said environment and the objects therein, and positioning sensors or cameras within said map.

[0003] The problem with the standard SLAM approach is that it is slow to process changes in the position or orientation of objects in the scene. For example, each time an object moves from one point to another, the entire scene needs to be scanned, and therefore all the points that define the object need to be scanned in order to find the new location and orientation of the object. This means that the entire scene needs to be scanned every time movement occurs, i.e. every point of the scene and every point and object point in the scene. This is disadvantageous because it is not an efficient process and consumes a lot of power. This is even more disadvantageous because it is a slow process, resulting in a lot of latency.

[0004] Another disadvantage is that if an object moves very quickly in the scene, standard SLAM methods cannot quickly find the new position and orientation of the object. Instead, there is a delay between the actual movement of the object and the moment the method finds the new position and orientation of the object. This can cause confusion to the user when using the method for augmented reality. In addition, if the object moves too quickly, this can also cause motion sickness because 1) the user perceives the object in a delayed manner from the actual movement, and 2) because the objects appear to change their positions quickly in an unrealistic manner. In addition, unless the object moves very slowly, it is impossible to see the object moving in a smooth manner in the scene.

[0005] Furthermore, for objects that move very quickly in the scene, scanning all points in the scene that define the object may result in an incomplete scan, as the object may move before the scan is complete. This may result in the user seeing only a portion of the object, or even worse: seeing an object in two different locations in the scene because the object moved before the scene scan was complete, again resulting in the user viewing a distorted, blurry, and confusing representation.

[0006] Therefore, there is a need for an improved SLAM method that is fast and efficient, and a system using the same.

[0007] The present invention is intended to address at least some of the problems mentioned above. Summary of the invention

[0008] An object of embodiments of the present invention is to provide low latency, fast, accurate and efficient sensing, for example, optical sensing, preferably for simultaneous localization and mapping (SLAM). The above objects are achieved by methods, sensors, devices and systems according to the present invention.

[0009] In a first aspect, the present invention relates to a method for sensing (e.g. optical sensing), preferably for simultaneous localization and mapping (SLAM), the method comprising the following steps:

[0010] a) identifying a second set of locations of points of at least one object in the scene within a second time window relative to a first reference point, such as preferably a first viewpoint;

[0011] as well as

[0012] b) converting the second set of locations into a second set of 3D positions,

[0013] Characterized in that the method further comprises the following steps:

[0014] c) determining a transformation to the second set of 3D positions such that the transformed second set of 3D positions best fits a first set of 3D positions, wherein the first set of 3D positions is denser than the second set of 3D positions (6), and wherein the first set of 3D positions was obtained in a first time window before the second time window,

[0015] And wherein the second time window is shorter than the first time window, preferably at least twice shorter.

[0016] Step (c) is preferably (but not necessarily) accomplished by calculating a third set of 3D positions, by determining a transformation of the second set of 3D positions, such that the third set of 3D positions best fits the first set of 3D positions.

[0017] An advantage of an embodiment of the present invention is that a low latency and efficient method for SLAM is obtained. An advantage of an embodiment of the present invention is that smooth movement of an object is obtained. An advantage of an embodiment of the present invention is that a user perceives the movement of an object in a scene with minimal or no delay. An advantage of an embodiment of the present invention is that motion sickness is prevented. An advantage of an embodiment of the present invention is that a user perceives the real movement of an object. An advantage of an embodiment of the present invention is that a user is prevented from having a blurred or distorted or incomplete view of an object in a scene.

[0018] An advantage of embodiments of the present invention is that only partial identification of the current location of the points defining an object is required. An advantage of embodiments of the present invention is that only partial identification of an object is required, thereby allowing low latency and efficient mapping and positioning of the object. An advantage of embodiments of the present invention is that although only partial identification of the object and the location of the points defining the object is required, accurate identification is still obtained. An advantage of embodiments of the present invention is that the need to identify all points defining an object is eliminated. An advantage of embodiments of the present invention is that not all points of an object need to be repeatedly identified in each instance. An advantage of embodiments of the present invention is that the points in the second 3D position set are much fewer than the points in the first 3D position set, preferably at least two times less, more preferably at least five times less, even more preferably at least ten times less, and still more preferably at least twenty times less, again more preferably at least fifty times less, and most preferably at least one hundred times less, so finding a transformed second 3D position set or a third 3D position set that best fits the first 3D point set is a fast, power efficient and low latency step. The method further facilitates reduced power consumption, since in the second time window, not all points defining the object need to be identified, ie the second set of 3D positions or the third set of 3D positions are sufficient to fit with the first set of 3D positions.

[0019] An advantage of embodiments of the present invention is that the unknown current location of points defining an object is obtained based on the previous locations of points defining the object. An advantage of embodiments of the present invention is that changes in the location of an object (i.e., the locations of points defining the object) that occur within a very short time window (e.g., shorter than 100 microseconds) can be identified.

[0020] An advantage of embodiments of the present invention is that continuous positioning of objects in the scene is obtained. An advantage of embodiments of the present invention is that quasi-continuous mapping of objects in the scene is obtained, "quasi" meaning that the update and positioning of the map is significantly faster than the movement of the objects and / or the observer.

[0021] An advantage of embodiments of the present invention is that the rotation of an object and points defining the object is calculated. An advantage of embodiments of the present invention is that points defining the object are adjusted based on a change in orientation of the object.

[0022] An advantage of the embodiments of the present invention is that the translation and rotation of the object between the first time window and the second time window are obtained.

[0023] An advantage of embodiments of the present invention is that, using sparse and dense data sets, a user's environment can be mapped with low latency and the user's gestures or head gestures can be recognized in a digital environment. This low latency is particularly advantageous in head mounted display applications.

[0024] Preferred embodiments of the first aspect of the present invention include suitable combinations of more than one of the following features.

[0025] The method preferably further comprises the following steps:

[0026] - identifying a first set of locations of points of said at least one object in said scene within said first time window relative to said first reference point, and

[0027] - converting said first set of places into said first set of 3D positions.

[0028] The method preferably further comprises the following steps:

[0029] - supplementing said second set of 3D positions with said first set of 3D positions.

[0030] An advantage of embodiments of the present invention is that the first set of 3D positions is reliable for supplementing the second set of 3D positions. An advantage of embodiments of the present invention is that the second set of 3D positions can be constructed based on the first set of 3D positions.

[0031] The method preferably further comprises the following steps:

[0032] - illuminating dots on the scene along an illumination trace, preferably in a Lissajous manner,

[0033] - monitoring said point-like object,

[0034] - identifying said first set of locations and said second set of locations along said track relative to said first reference point,

[0035] - identifying a third set of locations along said trace relative to a second reference point, preferably a second viewpoint, within said first time window,

[0036] - identifying a fourth set of locations along the track relative to the second reference point within the second time window,

[0037] - triangulating the first set of locations and the third set of locations to obtain the first set of 3D positions, and

[0038] - triangulating the second set of locations and the fourth set of locations to obtain the second set of 3D positions.

[0039] An advantage of an embodiment of the present invention is that the second set of locations (or the fourth set of locations) along the trail has much fewer points than the first set of locations (or the third set of locations) along the trail. An advantage of an embodiment of the present invention is that only points along the trail are identified, thereby allowing low-latency, fast and efficient mapping and positioning of the object in an accurate manner. An advantage of an embodiment of the present invention is that the trail covers, for example, at most 20%, preferably at most 10%, more preferably at most 5%, even more preferably at most 2%, and most preferably at most 1% of the scene during the second time window.

[0040] An advantage of embodiments of the present invention is that the need to illuminate the entire scene and identify all points of objects in the scene is eliminated. An advantage of embodiments of the present invention is that by only partially monitoring the scene, information about objects in all of the scene can be obtained. An advantage of embodiments of the present invention is that different areas of the scene are monitored equally or almost equally. An advantage of embodiments of the present invention is that different areas of the scene have an equal or almost equal chance of being scanned by the point-like object, especially in the Lissajous manner.

[0041] An advantage is that only one method (in this case only triangulation) is used to obtain the first set of 3D positions and the second set of 3D positions, as this allows for a simpler, faster and more efficient method compared to other methods that use two different methods to obtain dense and sparse 3D datasets.

[0042] The method preferably further comprises the following steps:

[0043] - defining a first set of line segments by consecutive points of said first set of 3D positions,

[0044] - defining a second set of line segments by consecutive points of said second set of 3D positions, and

[0045] - determining a transformation to said second set of line segments such that said second set of line segments best fits said first set of line segments.

[0046] An advantage of the embodiments of the present invention is that an accurate fit of the second 3D position set to the first 3D position set is obtained.

[0047] The method preferably further comprises the following steps:

[0048] - dividing said second time window into sub-windows, wherein each said sub-window comprises a subset of 3D positions, and

[0049] - for each of the 3D position subsets, calculating a corresponding transformed 3D position set, wherein each of the transformed 3D position sets is obtained by transforming the second 3D position set in the sub-window such that each transformed 3D position set best fits the first 3D position set.

[0050] An advantage of embodiments of the present invention is that changes in the location of points defining an object or rotations of said object occurring within a very short time window can still be identified. An advantage of embodiments of the present invention is that objects moving or rotating very quickly in said scene can still be monitored. An advantage of embodiments of the present invention is that the method allows for the flexibility of considering the entire second time window, or one or more sub-windows, depending on the need and the movement or rotation of said object in said scene.

[0051] The method preferably also includes the following steps:

[0052] - inferring a predicted set of 3D positions for an upcoming time instance based on the second set of 3D positions and the first set of 3D positions, wherein the inferring comprises identifying a trajectory of the object between the first time window and the second time window,

[0053] - identify a set of real 3D locations, and

[0054] - validating said predicted set by at least partly comparing it with said true set.

[0055] An advantage of embodiments of the present invention is that a set of 3D positions of objects of an upcoming detection is predicted. An advantage of embodiments of the present invention is that the movement of objects in a scene is predicted before it actually occurs. An advantage of embodiments of the present invention is that fast and low latency mapping is obtained while still obtaining a high accuracy mapping. An advantage of embodiments of the present invention is that inaccurate point identifications (e.g., false positives) are filtered out.

[0056] The method preferably further comprises the step of combining the first and second time windows and the points therein.An advantage of embodiments of the present invention is that denser data comprising many points defining the object within a longer time window is obtained.

[0057] In a second aspect, the present invention relates to an optical sensor for simultaneous localization and mapping (SLAM), comprising:

[0058] - a first plurality of pixel sensors, each pixel sensor comprising a photodetector, and

[0059] - processing unit,

[0060] wherein the photodetectors of the first plurality of pixel sensors are adapted to output a second set of locations of points of at least one object in the scene relative to the first plurality of pixel sensors within a second time window,

[0061] wherein the unit is adapted to convert the second set of locations into a second set of 3D positions,

[0062] characterised in that the unit comprises a processor which, in use, determines a transformation to the second set of 3D positions such that the second set of 3D positions best fits the first set of 3D positions,

[0063] wherein the first 3D position set is denser than the second 3D position set,

[0064] and wherein the first 3D position set is within a first time window before the second time window,

[0065] And wherein said second time window is shorter than said first time window, preferably at least two times shorter, and preferably wherein said first set of 3D positions is retrieved from a storage location, such as preferably a local or cloud-based storage location.

[0066] Preferred embodiments of the second aspect of the present invention include:

[0067] - said photodetector of said first plurality of pixel sensors is further adapted to output a first set of locations of points of said at least one object in said scene within said first time window, wherein said unit is further adapted to convert said first set of locations into said first set of 3D positions, and

[0068] - the sensor further comprises primary optics capable of producing an image of said scene on said first plurality of pixel sensors.

[0069] In a third aspect, the invention relates to an optical sensing device for simultaneous localization and mapping (SLAM), comprising the optical sensor according to the second aspect, wherein the device further comprises at least one light source, wherein the light source is adapted to illuminate at least one point on the scene along an illumination trace, preferably in a Lissajous manner,

[0070] wherein the apparatus further comprises a second plurality of pixel sensors, each pixel sensor comprising a photodetector,

[0071] wherein the photodetectors of the first plurality of pixel sensors are adapted to monitor the point-like object and output the first set of locations and the second set of locations along the track,

[0072] and wherein the photodetectors of the second plurality of pixel sensors are adapted to monitor the point-like object and output, relative to the second plurality of pixel sensors, a third set of locations of the object's points along the track within the first time window and a fourth set of locations of the object's points along the track within the second time window,

[0073] Wherein the unit is capable of triangulating the first set of locations and a third set of locations to obtain the first set of 3D positions, and wherein the unit is capable of triangulating the second set of locations and a fourth set of locations to obtain the second set of 3D positions.

[0074] Preferred embodiments of the third aspect of the present invention include suitable combinations of more than one of the following features:

[0075] - the device further comprises at least one memory element, wherein the element is adapted to store at least the first set of 3D positions and / or the second set of 3D positions,

[0076] The unit is capable of inferring a predicted set of 3D positions of an upcoming instance based on the second set of 3D positions and the first set of 3D positions, wherein the unit is capable of identifying a trajectory of the object between the first time window and the second time window, wherein the unit is also capable of verifying the predicted set by comparing it with a true set of 3D positions.

[0077] In a fourth aspect, the present invention relates to an optical sensing system for simultaneous localization and mapping (SLAM), comprising the optical sensing device according to the third aspect, wherein the system further comprises auxiliary optics capable of producing an image of the scene on the second plurality of pixel sensors.

[0078] The above and other characteristics, features and advantages of the present invention will become apparent from the following detailed description in conjunction with the accompanying drawings, which illustrate the principles of the present invention by way of example. This description is given for illustrative purposes only and does not limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The present disclosure will be further illustrated with the help of the following description and accompanying drawings.

[0080] Figure 1 (a, b) show a first set of locations (1) and a second set of locations (2) of points of an object (3) in 2D according to an embodiment of the present invention, and (c, d) show a first set of 3D locations (5) and a second set of 3D locations (6) of points of the object (3) in 3D.

[0081] Figure 2(a, b) show a first set of locations (1) and a second set of locations (2) in 2D of points of an object (3) relative to a first reference point (28) along an illumination trace (9) according to an embodiment of the present invention, and (c, d) show a third set of locations (11) and a fourth set of locations (12) in 2D of points of an object (3) relative to a second reference point (29), and (e, f) show a first set of 3D positions (5) and a second set of 3D positions (6) in 3D obtained by triangulating the first set of locations and the third set of locations (1, 11) and the second set of locations and the third set of locations (2, 12) respectively, and (g, h) show matching or fitting a third set of line segments (35) as a transformation of the second set of line segments (34) with the first set of line segments (33) so that the third set of line segments (35) best fits the first set of line segments (33).

[0082] Figure 3 In (a, b, c) objects (3) with different orientations according to an embodiment of the present invention are shown.

[0083] Figure 4 Points of an object (3) identified within four time windows (14, 15, 16, 17) of different widths are shown according to an embodiment of the present invention, wherein a dense data set (20) is obtained for the longest time window (17).

[0084] Figure 5 An object (3) is shown moving in a trajectory (19) between the first and second time windows (T1, T2) according to an embodiment of the present invention, wherein a set of predicted 3D positions (18) is obtained.

[0085] Figure 6 According to an embodiment of the present invention, (a) shows the identification of points along a Lissajous trace (10), wherein the points are used to define a second set of contour lines (30) in (b), and (c, e) shows the identification of points along a denser Lissajous trace (10), wherein the points are used to define a first set of contour lines (31) and a fourth set of contour lines (32) in (d, f).

[0086] Figure 7 An optical sensor (21) according to an embodiment of the present invention is shown.

[0087] Figure 8 An optical sensing device (26) according to an embodiment of the present invention is shown, which includes a first plurality of pixel sensors and a second plurality of pixel sensors (28, 29) and a light source (27), wherein the light source (27) illuminates a point-like object (8) on a scene (4) in a Lissajous manner (10).

[0088] Any reference signs in the claims should not be construed as limiting the scope.The same reference signs in different drawings refer to the same or similar elements. DETAILED DESCRIPTION

[0089] The present invention relates to an optical sensor and method for sensing (eg optical sensing), preferably for simultaneous localization and mapping (SLAM) with low latency.

[0090] The present invention will be described with respect to specific embodiments and with reference to certain drawings, but the invention is not limited thereto but only by the claims. The drawings described are only illustrative and non-restrictive. In the drawings, for illustrative purposes, the size of some elements may be exaggerated and not drawn to scale. Dimensions and relative dimensions do not correspond to actual practical reductions of the invention.

[0091] The terms "first", "second", etc. in the description and claims are used to distinguish similar elements and not necessarily to describe a sequence, temporally, spatially, in ranking, or in any other manner. It should be understood that the terms so used are interchangeable under appropriate circumstances, and that the embodiments of the invention described herein are capable of operation in other sequences than described or illustrated herein.

[0092] Furthermore, the terms "top", "lower", etc. in the description and claims are used for descriptive purposes and not necessarily for describing relative positions. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are capable of operation in other orientations than described or illustrated herein.

[0093] In the description provided herein, many specific details are set forth. However, it should be understood that embodiments of the present invention can be practiced without these specific details. In other cases, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0094] References throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the phrases "in one embodiment" or "in an embodiment" appearing in various places in this specification do not necessarily all refer to the same embodiment, but may be the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments, as will be apparent to one of ordinary skill in the art from this disclosure.

[0095] Similarly, it should be appreciated that in the description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together in a single embodiment, figure, or description thereof, for the purpose of simplifying the present disclosure and aiding in the understanding of one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than those expressly stated in each claim. On the contrary, as reflected in the following claims, inventive aspects lie in less than all the features of a single preceding disclosed embodiment. Therefore, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim independently serving as a separate embodiment of the present invention.

[0096] In addition, although some embodiments described herein include some features included in other embodiments but not other features, the combination of features of different embodiments is intended to be within the scope of the present invention and form different embodiments, as will be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments may be used in any combination.

[0097] Unless otherwise defined, all terms (including technical and scientific terms) used to disclose the present invention have the meaning commonly understood by one of ordinary skill in the art to which the present invention belongs. With the help of further guidance, term definitions are included to better understand the teachings of the present invention.

[0098] As used herein, the following terms have the following meanings: "a," "an," and "the," as used herein, refer to both singular and plural referents unless the context clearly indicates otherwise. As an example, "a pollutant" refers to one or more pollutants.

[0099] The recitation of numerical ranges by endpoints includes all numbers and fractions subsumed within that range, as well as the recited endpoints.

[0100] In a first aspect, the invention relates to a method for sensing, eg optical sensing, preferably for simultaneous localization and mapping (SLAM).

[0101] The method comprises the step of (a) identifying a second set of locations of points of at least one object in a scene within a second time window, the set comprising, for example, points defining the object, for example, points indicating that the object is present in a portion of the scene (e.g., identifying points defining an outline of the object).

[0102] The method further comprises the step (b) of converting the second set of locations into a second set of 3D positions. For example, the second set of 3D positions has depth information, such as xyz information about each point defining the at least one object.

[0103] The method further comprises a characterization step (c) determining a transformation to the second 3D position set such that the transformed second 3D position set best fits a first 3D position set, wherein the first 3D position set is denser than the second 3D position set, i.e., the first 3D position set includes more points than the second 3D position set. The first 3D position set was obtained in a first time window before the second time window.

[0104] Step (c) is preferably accomplished by calculating a third 3D position set, by determining a transformation of the second 3D position set, so that the third 3D position set best fits the first 3D position set. For example, the third 3D position set or the second 3D position set is fitted or matched to the first 3D position set, wherein the third 3D position set is a transformed version of the second 3D position set. This is not a point-to-point mapping, but a matching step of the entire second 3D position set or the third 3D position set with the entire first 3D position set, for example, matching one set of points with another set of points. For example, similar points of the object are allowed to be matched to each other as a group. The accuracy of the match or fit may be at least 80%, preferably at least 90%, more preferably at least 95%, and most preferably at least 98%.

[0105] The matching or fitting in the present invention is preferably accomplished by matching or fitting a sparse set of data points to a dense set of data points. This is because it is cheaper, easier, and faster to transform and fit a sparse set of points to a dense set of points than to transform and fit a dense set of points to a sparse set of points. This is also because a dense data set can in many cases represent the entire world (e.g., the entire digital world), while a sparse data set represents a portion of the world (e.g., the portion that the user is viewing). However, in principle, both approaches are possible.

[0106] Since the object does not move very fast between the first time window and the second time window, the match can be made with such high accuracy. For example, the time between the end of the first time window and the beginning of the second time window is usually 0 milliseconds, where the windows are adjacent, but the windows can also overlap in time (e.g., an overlap time of 100 microseconds to 1 millisecond), or there can be gaps between windows (e.g., between 100 microseconds and 1 millisecond).

[0107] The first set of 3D positions is preferably retrieved from a storage location, such as preferably a local or cloud-based storage location.

[0108] The second time window is shorter than the first time window, preferably at least twice shorter, or more than twice shorter. The ratio between the second time window and the first time window can be between 1:2 and 1:100, preferably between 1:5 and 1:50, and more preferably between 1:10 and 1:20. It is recognized by those skilled in the art that the ratio between the second time window and the first time window depends on the moving speed of the object and on the application. For example, there are more points identified in the first window than in the second window. For example, the sampling rate in the first window and the second window is the same, but because the length of the first time window is longer, there are more points identified in the first time window than in the second time window.

[0109] The width of the second time window is long enough to identify a small fraction of the number of points in the dense representation obtained from the first time window. The number of points in the second time window is at most equal to the number of points in the first time window, preferably at most 50%, more preferably 10%, most preferably at most 5%. For example, the second time window can be at most 5 milliseconds, preferably 1 millisecond, and more preferably at most 100 microseconds.

[0110] The second set of locations is identified relative to a first reference point, such as preferably a first viewpoint. The reference point may also be, for example, a location of a system performing the method, or, for example, a viewpoint of a sensor in the system. This allows identifying objects in the scene relative to the reference point, which allows mapping all objects in the scene, for example building a map of objects in the scene, preferably all objects in the scene, and further positioning the system performing the method in the scene. The at least one object is preferably a plurality of objects.

[0111] The method preferably further comprises the steps of identifying a first set of locations of points of the at least one object in the scene within the first time window relative to the first reference point, and converting the first set of locations into the first set of 3D positions. The first set of locations and the second set of locations are 2D (i.e., 2-dimensional), e.g., in an xy axis, e.g., each point has an x ​​coordinate and a y coordinate. The first set of locations is a dense data set, and the second set of locations is a sparse data set.

[0112] As mentioned above, step (c) is preferably performed by determining a transformation of the second set of 3D positions to obtain the third set of 3D positions. For example, determining a translation (i.e., a movement in the xyz direction) and / or a rotation (i.e., a change in orientation) of the second set of 3D positions corresponding to a translation and / or rotation of the at least one object to obtain the third set of 3D positions, thereby allowing a best fit to be obtained with the first set of 3D positions. For example, in order to obtain information about the movement and / or rotation of objects in the scene between the time window and the second time window. For example, in order to determine a change in orientation of the at least one object relative to a reference orientation or a reference axis system (e.g., an xyz axis system). For example, in the case where the orientation of the object changes relative to at least one axis.

[0113] Determining the translation and / or rotation is preferably done in an incremental manner. For example, an initial translation and / or rotation estimate is determined, after which the transformed second set of 3D positions or the third set of 3D positions is fitted or matched to the first set of 3D positions. However, in the event that the initial translation and / or rotation is unsuccessful, the matching or fitting process is continued by correcting or improving the translation and / or rotation estimate. This may be repeated until the matching is successful, or until a best match is obtained, or until a percentage error (e.g., a minimum error) is obtained. For example, a percentage error in matching the transformed second set of 3D positions or the third set of 3D positions to the first set of 3D positions may be tolerated.

[0114] Calculating the change in orientation (e.g., a rotation of an object) is important because at least some of the points defining the object may not be recognizable, e.g., they may disappear due to the rotation, e.g., they may be recognizable during the first time window but not during the second time window. On the other hand, at least some other points defining the object may appear due to the rotation, e.g., they may be recognizable during the second time window but not during the first time window. Therefore, calculating the orientation facilitates matching in an accurate manner, e.g., by adjusting the points defining the object based on the change in orientation, e.g., by adjusting points in the first set of 3D positions and / or the second set of 3D positions.

[0115] The method and transformation are preferably performed on a rigid body (e.g., a non-deformable body). For example, a box or a cylinder, which can be moved and / or rotated in the scene, but generally does not deform, i.e., does not change its shape. However, the invention is still applicable to non-rigid bodies, for example by identifying a new set of first locations and a new set of first 3D positions in a more frequent manner than in the case of a rigid body. For example, in the case of an inflated balloon, the transformation allows not only translational and / or rotational changes of the object, but also non-rigid deformations.

[0116] After determining the transformation, it is known where the object has moved and how it has rotated. Therefore, an object can be defined primarily based on the transformed second 3D position set or the third 3D position set, without having to identify all points of the object during the second time window. This allows fast, smooth and low-latency mapping and positioning of objects in the scene. In other words, using a sparse data set (e.g., the second 3D position set), the method can be used to map the user's environment and identify the user's location in the environment with ultra-low latency. Using the sparse data set, the user's head posture in the environment (e.g., a digital environment, such as the environment the user is looking at) can also be identified with ultra-low latency.

[0117] It is noteworthy that a sparse dataset can represent a scene (e.g., in scene coordinates), while a dense dataset can represent the world (e.g., in world coordinates). For example, considering a room or a given space, the dense data includes data representing the room (e.g., all objects of the room and, for example, walls, defined relative to a reference point in world coordinates). However, the sparse dataset includes data representing a scene or a portion of the room (e.g., the scene that a user of the method is looking at in scene or sensor coordinates). This is particularly advantageous for the following reasons. First, the sparse dataset is transformed and fit to the dense dataset, e.g., the method attempts to find the best fit for the scene in world coordinates, e.g., where the scene is located in the room. Second, finding the best fit allows finding the location of the user of the method in world coordinates, as well as the user's head pose (i.e., where the user is looking).

[0118] Preferably, the method further comprises the step of supplementing the second 3D position set with the first 3D position set. Based on the accuracy of the best fit in step (c), the supplementation step may also be accurate. This is advantageous for eliminating the need to identify missing points in the second 3D position set during the second time window, as they can be obtained and supplemented by the first 3D position set. In other words, supplementation allows the second 3D position set to be constructed based on the first 3D position set, which allows low latency and efficient SLAM.

[0119] Preferably, the ratio of the number of points in the second set of 3D positions to the number of points in the first set of 3D positions is between 1:2 and 1:50, preferably between 1:10 and 1:20.

[0120] Preferably, the method is performed continuously. This is advantageous to allow continuous positioning and mapping of the scene and the objects therein.

[0121] Preferably, the method step may further comprise the step of illuminating a point-like object on the scene along an illumination trace. For example, the point-like object is illuminated by a light source (e.g. a point source). For example, a mechanical device (e.g. a MEMS mirror) moves the point-like object to illuminate different parts of the scene. The illumination trace is preferably continuous, but not necessarily, for example it may also be turned on and off.

[0122] The method steps may also include the step of monitoring the point-like object, for example, by a sensor viewing the scene.

[0123] The method steps may further include the step of identifying the first set of locations and the second set of locations along the track relative to the first reference point. The method steps may further include the step of identifying a third set of locations along the track relative to the second reference point within the first time window. The second reference point is preferably a second viewpoint. The method steps may further include the step of identifying a fourth set of locations along the track relative to the second reference point within the second time window. The difference between the third set of locations and the first set of locations is that they are identified from different reference points (e.g., the second reference point and the first reference point). For example, both the first set of locations and the third set of locations are identified within the first time window, and since the first time window is longer than the second time window, the first set of locations and the third set of locations have much more points than the second set of locations and the fourth set of locations. The fourth set of locations also differs from the second set of locations in that they are identified from different reference points (e.g., the second reference point and the first reference point).

[0124] The method steps may also include the step of triangulating the first set of locations and the third set of locations to obtain the first set of 3D locations. The method steps may also include the step of triangulating the second set of locations and the fourth set of locations to obtain the second set of 3D locations. The first set of locations and the third set of locations allow triangulation to obtain the first set of 3D locations. For example, by monitoring the point-like objects and identifying points along the trail, points of one set of locations at one reference point may be associated with corresponding points in another set of locations at another reference point. This allows depth information about objects in the scene to be obtained.

[0125] The lighting trace is also advantageous because only the points along the trace are identified within the second time window, which allows low latency mapping, e.g. not all points in the scene are identified. The trace also covers only a portion of the scene during the second time window, e.g. only at most 20% or at most 10% of the scene, and preferably at most 5%, and more preferably at most 2%, and most preferably at most 1%. Similarly, in future time windows, only points along the trace need to be identified, since only occasionally points along a longer time window (e.g., the first time window) need to be identified.

[0126] Preferably, future time windows (e.g., the third time window, the fourth time window, and the fifth time window) and the location sets identified by these time windows can be combined to obtain a new location set, which includes many points defining the object in a longer time window. The new location set can then be converted into a 3D position and used to map the 3D position in the future time window.

[0127] Preferably, the first set of 3D positions may be updated periodically. For example, the first set of 3D positions may be updated after a predetermined period of time, for example by identifying a new set of points within a third time window adapted to be long enough (similar to the first time window) to identify many points of an object (e.g., a new set of dense 3D points) so as to allow matching or fitting a new set of sparse 3D points to the new set of dense 3D positions. In other words, after each predetermined period of time, a new dense data set may be obtained and used to match or fit a new sparse data set.

[0128] Preferably, the point-like object can be sparsely illuminated on the scene. For example, only certain parts of the scene are illuminated, and the points are identified along the illumination trace caused by the point-like object within a predetermined time window. For example, the certain parts are illuminated in such a way that the points of the object in different parts of the scene (for example, any part of the scene) can be identified. However, the identification of the points is performed in a rough manner, for example, only the points of the object in the scene are partially identified, for example, if the object is defined by 100 points, only a part of these points (for example, at most 5 points or at most 10 points or at most 20 points) are identified. By identifying these points, the overall shape or outline of the object can be identified in a rough manner. In the previous time window, the point-like object is scanned more intensively, wherein more points of the object are identified, for example, for an object with 100 points, at least 80 points are identified, for example, at least 90 points, for example, at least 95 points, for example, at least 98 points. By first converting the points from 2D to 3D, and mapping the previous set of points defining the object (e.g., including 5 or 10 points) to a later set of points (e.g., including 95 or 98 points), supplementary points can be obtained based on the latter set (i.e., dense set) to supplement the previous set (i.e., sparse set). It is worth noting that the supplementary points are based on the latter set (i.e., dense set), but are not necessarily the exact points obtained from the latter set (i.e., sparse set). For example, if the object is rotated, some of the supplementary points may have to be adjusted.

[0129] Preferably, the point-like object can be illuminated in a Lissajous manner, because the manner allows the points defining an object to be roughly identified without having to identify each individual point of the object every time. The Lissajous manner is an example, but the same effect can be obtained by other sparse illuminations (such as raster scanning). This is advantageous for obtaining a low-latency and efficient method for SLAM. This is also advantageous for allowing any object in any point of the scene to be identified, for example, since the point-like object has a similar scanning pattern in different parts of the scene, the object on the right side of the scene and the same object on the left side of the scene may have a similar number of identification points. Other advantages of the Lissajous manner are shown below and in the figures.

[0130] Preferably, the method may further comprise the step of defining a first set of line segments by continuous points of the first set of 3D positions. The method may further comprise the step of defining a second set of line segments by continuous points of the second set of 3D positions. For example, the continuous points may be continuous points on a Lissajous trace, where, for example, each line segment intersects the object once. This is also Figure 2Since the line segment is a 3D line segment, it is obvious that the line segment may be a straight line segment or may not be a straight line segment.

[0131] The method may also include the step of determining a transformation of the second set of line segments so that the second set of line segments best fits the first set of line segments. Finding the best fit or match between two sets of line segments may be more accurate and / or easier than finding the best fit or match between two sets of points. This may be accomplished by calculating a third set of line segments, wherein the third set of line segments is a transformed version of the second set of line segments.

[0132] Thus, this step may comprise determining a transformation of the second set of line segments to obtain the third set of line segments such that the third set of line segments best fits the first set of line segments. For example, determining a translation and / or rotation of the second set of line segments corresponding to a translation (i.e., movement in xyz) and / or rotation (i.e., change in orientation) of the at least one object to obtain the third set of line segments allows obtaining a best fit or match with the first set of line segments.

[0133] Preferably, the method may further comprise the step of dividing the second time window into sub-windows, wherein each of the sub-windows comprises a 3D position subset, such as a 3D position subset corresponding to a location subset identified in each of the sub-windows. The method may further comprise the step of calculating a corresponding transformed 3D position set for each of the 3D position subsets, wherein each of the transformed 3D position sets is obtained by transforming the second 3D position set in the sub-window such that each transformed 3D position set best fits the first 3D position set.

[0134] Dividing the second time window into sub-windows allows obtaining information about very fast moving or rotating objects in the scene, for example by obtaining information about the objects in each of the sub-windows. For example, by fitting each of the 3D position subsets of each of the sub-windows to the first set of 3D positions. Depending on the speed of the movement or rotation of the objects, the 3D positions within one or more of the sub-windows may be considered for the fitting.

[0135] Preferably, the number of points in the 3D position subset within a very short sub-window may need to be higher, for example by identifying more points of the object in the sub-window. For example, the 3D position subset in a very short sub-window may be fitted to the first set of 3D positions, as long as the 3D position subset has a sufficient number of points to perform such a fit.

[0136] Preferably, the method may further include combining the second 3D position set with a fourth 3D position set (e.g., wherein the fourth 3D position set is within a third time window), for example, if the second 3D position set does not have enough points to fit into a dense 3D position set (e.g., the first 3D position set). Alternatively, the method may further include combining the first time window with the second time window and points thereof, or combining any other time windows with each other.

[0137] Preferably, the method may also include a step of storing a dense data set (e.g., the first 3D position set and / or the first location set) in at least one memory element. For example, in order to use the dense data set to fit or match it. For example, the dense data set is an aggregation of a set of locations in a series of time windows, wherein the number of the series of time windows is sufficient to allow the dense data set to include, for example, at least 50% of the points of at least one object in the scene, preferably at least 80% of the points, more preferably at least 90% of the points, and most preferably at least 95% of the points. The step may also include storing sparse data sets or other data. Such storage can be done locally or in a cloud-based storage location.

[0138] Preferably, the method may further comprise the step of updating the dense data set. For example, in a time window much later than the first time window, a new dense data set may be obtained, such as a new first set of locations, for example especially if the object has been significantly moved and / or rotated. The same steps of obtaining a new set of locations are performed, followed by updating the dense data set, followed by performing further fitting or matching and triangulation based on the new first set of locations or new set of 3D positions.

[0139] Preferably, the method steps include the steps of defining a first set of contour lines by the outermost points of the first set of 3D positions and defining a second set of contour lines by the outermost points of the second set of 3D positions. The contour is defined, for example, by connecting the outermost points of the first set of 3D positions or the second set of 3D positions to each other to form the contour of the object. The second set of contour lines is then transformed into the third set of contour lines by determining a transformation, so that the third set of contour lines or the transformed second set of contour lines matches or fits the first set of contour lines. Since the contour lines are defined using a larger set of 3D positions, the contour lines may be more advantageously and accurately used for fitting than a set of points. Therefore, the method steps preferably include the step of fitting the second set of contour lines to the first set of contour lines.

[0140] As described above, using Lissajous trails is advantageous for defining the second set of locations in a better manner, because after several or several Lissajous line segments, many points in the second set of locations are identified, and based on these points the second set of 3D positions and subsequently the second set of contour lines are defined, wherein the more Lissajous line segments there are and the longer the identification window of the second set of locations, the better the second set of contour lines is defined. Figure 6 This embodiment is better illustrated in Alternatively, the first and second sets of contour lines may be defined by the first and second sets of locations in 2D, after which the first and second sets of contour lines are converted into 3D.

[0141] The contour of this embodiment describes the line segments that define the outermost points of an object, e.g., the shape of the object, such as Figure 6 This allows object segmentation, for example by learning the location of each object, for example by drawing a set of contour lines around the object.

[0142] Preferably, the method steps may also include the step of inferring a predicted 3D position set for an upcoming time instance or time window based on the second 3D position set and the first 3D position set, wherein the inference includes identifying the trajectory of the object between the first time window and the second time window. For example, if the trajectory of the object moves in the correct direction between the first time window and the second time window, it is still expected to move further in the correct direction in the upcoming time instance. In addition, the moving speed of the object between the first time window and the second time window can be estimated, which is used to better infer the predicted 3D position set within the upcoming time instance or time window.

[0143] This is advantageous for predicting the movement of the object outside the first time window and the second time window before it actually occurs. The method steps may also include the step of identifying a real 3D position set. The real 3D position set may only include a few points of the object, for example, the real 3D position set corresponds to a partial identification of the points of the object. The method steps may also include the step of verifying the predicted set by at least partially comparing with the real set. The real 3D position set is only used for verification purposes. For example, to verify whether the object has not changed its trajectory or movement speed.

[0144] This further helps filter out false positives and inaccurate point identifications. For example, if an object is confirmed to have moved in a certain direction at a certain speed, then the identification of a point far from the predicted 3D position set may be a false positive unless it is related to another object in the scene.

[0145] In a second aspect, the present invention relates to an optical sensor for SLAM. The system includes a first plurality of pixel sensors, each of which includes a photodetector. The photodetector of the first plurality of pixel sensors is suitable for outputting a second set of locations of a point of at least one object in a scene within a second time window relative to the first plurality of pixel sensors. For example, relative to a point in the middle of the first plurality of pixel sensors.

[0146] The sensor further comprises a processing unit. The unit is adapted to convert the second set of locations into a second set of 3D positions. Thereafter, the second set of 3D positions can be stored in a random access memory of the unit and retrieved when needed.

[0147] The unit comprises a processor which, when in use, determines (e.g., is capable of determining) a transformation of the second set of 3D positions so that the second set of 3D positions best fits the first set of 3D positions. For example, a translation and / or rotation of the second set of 3D positions corresponding to a translation and / or rotation of an object in the scene is determined relative to the first plurality of pixel sensors. This is preferably performed continuously. As shown in the method of the first aspect, the transformation is advantageous for allowing the following: a low latency and high efficiency method is obtained, and subsequently a low latency and high efficiency sensor is obtained. Therefore, the sensor is able to map objects in the scene and position the first plurality of pixel sensors relative to the objects. The sensor may include an instruction set which, when executed by the processor, performs the method of the first aspect.

[0148] The first set of 3D positions is denser than the second set of 3D positions. The first set of 3D positions is within a first time window before the second time window. The second time window is shorter than the first time window, preferably at least twice shorter. The first set of 3D positions is preferably retrieved from a storage location, such as preferably a local or cloud-based storage location.

[0149] Preferably, the photodetectors of the first plurality of pixel sensors are further adapted to output a first set of locations of points of the at least one object in the scene within the first time window, wherein the unit is further adapted to convert the first set of locations into the first set of 3D positions.

[0150] Preferably, the unit is capable of supplementing said second set of 3D positions with said first set of 3D positions. This allows building up said second set of 3D positions, after which said unit can be connected to means for producing an image of said scene.

[0151] Preferably, the pixel sensors can be arranged in multiple rows and / or multiple columns. For example, the pixel sensors can be arranged in a matrix configuration. This arrangement allows detection of a larger area. Preferably, each photodetector can be a single photon detector, such as a single photon avalanche diode (SPAD).

[0152] Preferably, each detector can be arranged in a reverse biased configuration. The detector is capable of detecting photons impinging thereon. The detector has an output terminal for outputting an electrical detection signal after detecting a photon. For example, a detection signal can be represented by a signal comprising a logic "1" (e.g., detection), and a non-detection signal can be represented by a signal comprising a logic "0" (e.g., no detection). Alternatively, a detection signal can be represented by a pulse signal or cause a pulse signal to transition, for example, from a logic "0" to a logic "1" and then from a logic "1" back to a logic "0", and a non-detection can be represented by the absence of such a pulse signal or cause the absence of such a pulse signal.

[0153] Preferably, the number of points in the first 3D position set is at least twice as many as the number of points in the second 3D position set, preferably at least five times, more preferably at least ten times, even more preferably at least twenty times, most preferably at least fifty times. This is because the first time window is longer than the second time window, e.g. more points of the object have been identified in the first time window.

[0154] Preferably, the sensor further comprises primary optics capable of producing an image of said scene on said first plurality of pixel sensors.

[0155] In a third aspect, the present invention relates to an optical sensing device for SLAM, comprising the optical sensor according to the second aspect.

[0156] The device also includes at least one light source. The light source is suitable for illuminating at least one point on the scene along the illumination trace. The device also includes a second plurality of pixel sensors, each pixel sensor including a photodetector. The second plurality of pixel sensors are preferably the same as the first plurality of pixel sensors. For example, the first plurality of pixel sensors and the second plurality of pixel sensors preferably have the same number of pixel sensors in the same construction. However, the first plurality of pixel sensors and the second plurality of pixel sensors are not located at the same position. For example, the first plurality of pixel sensors are offset from the second plurality of pixel sensors along the x-axis. This allows the outputs of the first plurality of pixel sensors and the second plurality of pixel sensors to be triangulated to obtain depth information of the scene, as will be explained below. Preferably, the light source is suitable for being at a wavelength detectable by the pixel sensors. For example, between 100 nanometers and 10 micrometers, preferably between 100 nanometers and 1 micrometer.

[0157] The photodetector of the first plurality of pixel sensors is adapted to monitor the point-like object. The detector is further adapted to identify and output the first set of locations and the second set of locations along the track. Similarly, the photodetector of the second plurality of pixel sensors is adapted to monitor the point-like object and output a third set of locations of the object's points along the track within the first time window and a fourth set of locations of the object's points along the track within the second time window relative to the second plurality of pixel sensors.

[0158] The unit is capable of triangulating the first set of locations and a third set of locations to obtain the first set of 3D positions, and wherein the unit is further capable of triangulating the second set of locations and a fourth set of locations to obtain the second set of 3D positions.

[0159] The light source is advantageous for allowing triangulation. For example, using the light source illuminated on a scene (e.g., an environment), and the first plurality of pixel sensors and the second plurality of pixel sensors are, for example, oriented differently from each other, wherein the first plurality of pixel sensors and the second plurality of pixel sensors have a shared field of view of the scene, the xy time data of the two sensors can be converted into xyz time data by triangulation. For example, by adjusting the light source to illuminate a point-like object on the scene in an illumination trace, wherein the point-like object scans the scene (preferably continuously), wherein the first plurality of pixel sensors and the second plurality of pixel sensors monitor the point-like object and output in multiple instances the location of a point of at least one object in the scene (e.g., a point of the surface of the object) along the trace, wherein the xy time data of the first plurality of pixel sensors and the second plurality of pixel sensors can be converted into xyz time data using triangulation. The light source can, for example, act as a reference point so that the location of the point of the object of the first plurality of pixel sensors can be synchronized with the location of the point of the object of the second plurality of pixel sensors to create a z dimension.

[0160] Preferably, the 3D position set can be filtered, for example, to reduce the number of pixels with false detections, and only the data of pixels with true detections are triangulated. For example, the detection of adjacent sensing units (e.g., pixels) is checked, or the persistence of the detection over time is checked, for example, as described in PCT IB2021 054688, PCT EP2021 087594, PCTIB2022000323 and PCT IB2022 058609. This is a pixel-by-pixel filtering mechanism. Another filtering mechanism can also be obtained, for example, after projecting the detection to the periphery, some noise detections can be removed because the projection pattern is known, such as Lissajous, etc. For example, filtering can be based on expected pulse width, expected point projection size, (one or more) past detections of the pixel and (one or more) current detections and (one or more) neighboring pixels. Thereafter, the detected signal is output on the row signal bus and the column signal bus, thereby allowing the location (i.e., x, y coordinates) of the laser point at each timestamp to be known. For example, filtering is performed based on past bus information to identify detection locations with the highest likelihood of having detected photons originating from active illumination.

[0161] Preferably, the light source advantageously illuminates and scans the scene to be imaged in a Lissajous manner or pattern. Such an illumination trajectory is advantageous for allowing efficient and rapid image detection, since after several illumination cycles a large portion of the image has been illuminated. Other illumination patterns, such as raster scanning, etc., can be envisioned.

[0162] Preferably, the first plurality of pixel sensors and / or the second plurality of pixel sensors may include more than 100 sensors, preferably more than 1000 sensors, more preferably more than 10000 sensors, even more preferably more than 100000 sensors, most preferably more than 1000000 sensors. For example, the first plurality of pixel sensors and / or the second plurality of pixel sensors may be arranged in a matrix, wherein the first plurality of pixel sensors and / or the second plurality of pixel sensors include 1000 pixel sensor rows and 1000 pixel sensor columns.

[0163] Preferably, the device further comprises at least one memory element, wherein the element is adapted to store at least the first 3D position set and / or the second 3D position set. For example, wherein the first 3D position set has at least twice as many data points of the object as the second 3D position set. The memory element may also store other data, such as the first location set and / or the second location set and / or the third 3D location set.

[0164] Preferably, the unit is capable of inferring a predicted 3D position set of an upcoming instance based on the second 3D position set and the first 3D position set, wherein the unit is capable of identifying the trajectory of the object between the first time window and the second time window, and wherein the unit is also capable of verifying the predicted set by comparing with the real 3D position set.

[0165] In a fourth aspect, the present invention relates to an optical sensing system for SLAM, comprising the optical sensing device according to the third aspect. The system further comprises auxiliary optics capable of producing an image of the scene on the second plurality of pixel sensors.

[0166] The image includes, for example, the at least one object in the scene. The system is capable of constructing an image of the scene with low latency because the sensor is capable of constructing a second 3D position set including several data points of the object based on a first 3D position set including many data points of the object, as shown in the first and second aspects.

[0167] Preferably, the system may be used for 3D vision applications. For example, the system is used to visualize an object in three dimensions. For example, the at least one optical sensor may allow said vision.

[0168] Any features of the second aspect (ie, sensor), the third aspect (ie, device) and the fourth aspect (ie, system) may be as correspondingly described in the first aspect (ie, method).

[0169] In a fifth aspect, the present invention relates to using the method according to the first aspect, and / or the sensor according to the second aspect, and / or the device according to the third aspect, and / or the system according to the fourth aspect, for sensing (e.g. optical sensing), preferably for SLAM.

[0170] Further characteristics and advantages of embodiments of the invention will be described with reference to the drawings.It should be noted that the invention is not limited to the specific embodiments shown in the drawings or described in the examples, but only by the claims.

[0171] Figure 1 In (a, b) a first set of locations (1) and a second set of locations (2) of points of an object (3) are shown. The sets of locations (1, 2) are identified in 2D. The first set of locations (1) is identified within a first time window (T1). The locations (1) are relative to a first reference point (28), wherein the first reference point (28) is, for example, a system performing the method of the present invention, or, for example, a camera of the system. The second set of locations (2) is identified within a second time window (T2) later than the first time window (T1). Figure 1It is also shown that the first time window (T1) is much longer than the second time window (T2). This allows the first set of locations (1) to be much larger than the second set of locations (2). For example, more points are identified in the first set of locations (1) than in the second set of locations (2) because the identification time, i.e., in the first time window (T1) is much greater than in the second time window (T2). The set of locations (1, 2) is then converted from 2D to 3D, as Figure 1 As shown in (c, d) of . For example, the first set of locations (1) is converted into a first set of 3D locations (5), and the second set of locations (2) is converted into a second set of 3D locations (6). Thereafter, the second set of 3D locations (6) is fitted or matched with the first set of 3D locations (5). For example, by transforming the second set of 3D locations (6) into a third set of 3D locations (7), the third set of 3D locations (7) is fitted or matched with the first set of 3D locations (5), in such a way that the third set of 3D locations (7) fits the first set of 3D locations (5) in the best possible manner. The fitting or matching is done at the level of the entire set of points, rather than by point-by-point mapping. Therefore, the number of points in the second set of 3D locations (6) should be large enough to allow correct and accurate mapping (e.g., by having a sufficiently large second time window (T2)), but small enough to allow low latency and fast mapping processing. After the matching or fitting is successfully completed, the transformation required to obtain such a successful fit is determined. For example, how far the second 3D position set is relative to the first 3D position set (ie, how much the object has moved), and how much the orientation of the second 3D position set has changed relative to the first 3D position set (ie, how much the object has rotated).

[0172] After mapping is successfully completed, the second set of 3D positions (6) is supplemented by the first set of 3D positions (5), for example, the first set of 3D positions (5) allows the second set of 3D positions (6) to be as complete as the first set of 3D positions (5).

[0173] Figure 2In (a, b) a first set (1) and a second set (2) of locations of points of an object (3) are shown. A moving point-like object (8) generated by a light source (27) is illuminated on a scene (4) along an illumination trace (9), which in this case is a Lissajous pattern (10), but can also be other patterns. The density of line segments of the trace (9) in the scene (4) is related to the time window in which the point-like object (8) has moved in the scene (4). The more times the moving point-like object (8) scans the scene (4), the greater the density of line segments, and the more points of the object (3) are identified along the trace (9) of the point-like object (8). As shown in the figure, the number of points along the trace (9) in the first time window (T1) is greater than the number of points along the trace (9) in the second time window (T2), simply because the time window (T1) is longer than the second time window (T2). The first set of locations and the second set of locations (1, 2) are identified relative to a first reference point (28). The third set of locations (11) and the fourth set of locations (12) are identified relative to a second reference point (29), the second reference point (29) being different from the first reference point (28), for example, the first reference point and the second reference point (28, 29) being far from each other. Figure 2 As shown in (c, d), the third location set and the fourth location set (11, 12) show the orientation of the object (3) and Figure 2 The first set of locations (a, b) is slightly different because the reference points are different. By triangulating the first set of locations and the third set of locations (1, 11), a first 3D position set (5) is obtained, as shown in FIG. Figure 2 Similarly, by triangulating the second set of locations and the fourth set of locations (2, 12), a second set of 3D locations (6) is obtained. Triangulation allows the depth information of the set of locations (1, 2, 11, 12) to be obtained to obtain the first set of 3D locations and the second set of 3D locations (5, 6). Figure 1 Similar to what is explained in , the second 3D position set (6) is matched with the first 3D position set (5).

[0174] Figure 2(g) shows another matching or fitting method, which is based on a first line segment set (33) of the first 3D position set (5) and a second line segment set (34) of the second 3D position set (6). The first line segment set (33) is a dense line segment set defined by a dense data point set of the object (3). The first line segment set (33) includes line segments formed by connecting consecutive points of the first 3D position set (5). Similarly, the second line segment set (34) is obtained. Thereafter, a third line segment set (35) is calculated by transforming the second line segment set (34) so ​​that the third line segment set (35) fits or matches the first line segment set (33) in the best possible way. Matching two line segment sets can be easier, faster and more accurate than matching two groups of points. In addition, the second line segment set or the third line segment set (34, 35) can be supplemented by the first line segment set (33).

[0175] Figure 3 An object (3) is shown in 3D with different orientations. The orientation is calculated relative to a reference orientation (e.g. relative to a system performing the method). Calculating the orientation is useful for matching a third set of 3D positions (7) with the first set of 3D positions (5). Calculating the orientation is important because at least some points in the second set of 3D positions (6) may be hidden due to the change in orientation, in which case the matching of the second or third set of 3D positions (6, 7) with the first set of 3D positions (5) should be adjusted accordingly.

[0176] Figure 4 Points of an object (3) identified in four time windows (14, 15, 16, 17) with different widths are shown. The time window (14) is the shortest, while the time window (17) is the longest. Obviously, the longer the time window, the more points are identified. For a sufficiently long time window (e.g., the longest time window (17)), a dense data set (20) is obtained. After the dense data set (20) is converted into 3D, for example, by triangulation (as shown above), it is used as an ideal data set for the sparse data set to be matched therewith, and for providing a supplementary 3D position. However, 3D data obtained in a shorter time window can also be used for matching, for example, a 3D data set obtained in a time window (15) can be matched with a 3D data set obtained in a time window (16), and (14) is matched with (16) and (14) is matched with (15). In other words, it is not always necessary to match a sparse data set with a dense data set, that is, it may be possible to match a semi-sparse data set with a semi-dense data set, or to match a sparse data set with a denser data set.

[0177] Figure 5An object (3) is shown moving in a trajectory (19) between a first time window (T1) and a second time window (T2). As explained above, a third set of 3D positions (7) is mapped to the first set of 3D positions (5), which can then be used as a supplementary 3D position (7) to supplement the second set of 3D positions (6). It is then desired to find the location of the object (3) and the points defining the object (3) at an upcoming time instance (T3) (i.e. in the near future, e.g. 1 millisecond later). Based on the movement trajectory (19) of the object (3) in the past (e.g. between the first time window (T1) and the second time window (T2)), a predicted set of 3D positions (18) is inferred. Not only the movement of the object (3) itself, but also its rotation and orientation changes are taken into account to obtain the predicted set of 3D positions (18). For verification purposes, the real set of 3D positions of the object (3) at the upcoming time instance (T3) is identified (e.g. partially identified) so that the predicted set can be verified by comparison with the real set. It is noteworthy that the extrapolation of the predicted 3D position set (18) works better when the predicted 3D position set (18) is based on more 3D position sets (e.g., at least two 3D position sets, e.g., five 3D position sets, e.g., ten 3D position sets) (e.g., where the identification of the points of the object is done over a longer period of time), as this will enable the trajectory (19) to be predicted in a more accurate manner.

[0178] Figure 6 Identifying points along the Lissajous trace (10) is shown in (a). The points are converted to 3D and then used to define a second set of contour lines (30) in (b, g).

[0179] Although in Figure 6 Only some points of the object (3) are identified in (a), but the second set of contour lines (30) indicates the object. Figure 6 In (c) the same as in (a) is shown, but with a denser Lissajous trace (10), for example, where the identification of the points along the trace (10) is done over a longer window in (c) than in (a). By identifying the points defining the object (3) with the denser Lissajous trace (10) in (c) and converting them into 3D, a first set of contour lines (31) is defined in (d, h) that gives a more ideal representation of the object (3) than the second set of contour lines (30). In (e) another Lissajous trace (10) is shown, where a fourth set of contour lines (32) is defined in (f, i).

[0180] Then the second set of contour lines (30) is matched or fitted to the first set of contour lines (31) or the fourth set of contour lines (32). Figure 6 As shown in (a, b), this matching allows only roughly identifying some points of the object (3) in the second contour line set (30) is sufficient to identify the movement and / or rotation of the object (3) between different time windows.

[0181] Figure 7 An optical sensor (21) is shown, comprising a first plurality (28) of pixel sensors (22), each pixel sensor (22) comprising a photodetector (23). The optical sensor (21) further comprises a processing unit (24) and a memory element (25). Photons impinge on the photodetector (23), wherein the photodetector (23) is adapted to output a first set (1) and a second set (2) of locations of an object (3) in a scene (4). Figure 8 In the optical sensor (21) is shown as part of an optical sensing device (26) in which there are a first plurality (28) of pixel sensors (22) and a second plurality (29) of pixel sensors (22) and a light source (27). The device (26) allows triangulation of a first set of locations (1) obtained by the first plurality (28) of pixel sensors with a third set of locations (11) obtained by the second plurality (29) of pixel sensors, and triangulation of a second set of locations (2) obtained by the first plurality (28) of sensors with a fourth set (12) obtained by the second plurality (29) of pixel sensors, in order to obtain the first set of 3D positions and the second set of 3D positions (5, 6). This is accomplished by illuminating the light source (27), wherein the light source (27) illuminates a point (8) on the scene (4) in a Lissajous manner (10), and wherein the first plurality of pixel sensors and the second plurality of pixel sensors (28, 29) monitor the point (8) and identify the set of locations (1, 2, 11, 12) along the trail of the point (8), in this case along the Lissajous trail (10). It is noteworthy that the first plurality of pixel sensors and the second plurality of pixel sensors (28, 29) are not in the same position, for example, they are displaced from each other on the x-axis and / or y-axis, such as Figure 8 to allow the triangulation.

[0182] Other arrangements for achieving the purpose of implementing the methods and apparatus of the present invention will be apparent to those skilled in the art. The foregoing description gives details of certain embodiments of the present invention. However, it will be clear that no matter how detailed the above text is, the present invention can be applied in many ways. It should be noted that the use of certain terms in describing certain features or aspects of the present invention should not be interpreted as implying that the terms herein are again defined to be limited to the specific features or aspects of the present invention associated with the terms.

[0183] Reference Mark List

[0184] T1 first time window T2 second time window 1 first location set

[0185] 2 Second location set 3 Object 4 Scene

[0186] 5First 3D position set 6Second 3D position set

[0187] 7 Third 3D position set 8 Point objects 9 Lighting traces

[0188] 10 Lisaru 11 Gathering at the third location 12 Gathering at the fourth location

[0189] 14,15,16,17 Four time windows with different widths 18 Predicted 3D position set 19 Trajectory 20 Dense dataset 21 Optical sensor

[0190] 22 Pixel sensor 23 Photodetector 24 Processing unit

[0191] 25 memory element 26 optical sensing device 27 light source

[0192] 28 first reference point or first plurality of pixel sensors

[0193] 29 second reference point or second plurality of pixel sensors 30 second contour line set 31 first contour line set 32 ​​fourth contour line set

[0194] 33 The first line segment set 34 The second line segment set 35 The third line segment set

Claims

1. A method for sensing, comprising the following steps: a) identifying a second set of locations (2) of points of at least one object (3) in a scene (4) within a second time window (T2) relative to a first reference point (28), such as preferably a first viewpoint; wherein the second set of locations (2) is in 2D, and b) converting the second location set (2) into a second 3D position set (6), wherein the method further comprises the following steps: c) determining a transformation to the second set of 3D positions (6) such that the transformed second set of 3D positions (6) best fits the first set of 3D positions (5), wherein the first set of 3D positions (5) corresponds to the first set of locations (1), wherein the first set of locations (1) is in 2D, wherein the first set of 3D positions (5) is denser than the second set of 3D positions (6), and wherein the first set of 3D positions (5) is obtained in a first time window (T1) before the second time window (T2), and wherein the second time window (T2) is shorter than the first time window (T1), preferably at least twice shorter, Each of the first set of 3D positions (5) and the second set of 3D positions (6) is obtained by triangulating two sets of locations in 2D relative to two different reference points (28, 29).

2. The method according to claim 1, wherein the method comprises the following steps: - identifying said first set (1) of locations of points of said at least one object (3) in said scene (4) within said first time window (T1) relative to said first reference point (28), and - converting said first set of places (1) into said first set of 3D positions (5).

3. The method according to any one of the preceding claims, wherein the method further comprises the following steps: The second set of 3D positions (6) is supplemented with supplementary points based on the first set of 3D positions (5).

4. The method according to any one of the preceding claims, wherein the method further comprises the following steps: - illuminating a point (8) on the scene (4) along an illumination trace (9), - monitoring said point (8), - identifying said first set of locations and said second set of locations (1, 2) along said track (9) relative to said first reference point (28), - identifying a third set of locations (11) along said track (9) within said first time window (T1) relative to a second reference point (29), said second reference point (29) being preferably a second viewpoint, - identifying a fourth set of locations (12) along said track (9) relative to said second reference point (29) within said second time window (T2), - triangulating the first set of locations and the third set of locations (1, 11) to obtain the first set of 3D positions (5), and - triangulating the second set of locations and the fourth set of locations (2, 12) to obtain the second set of 3D positions (6).

5. The method according to any one of the preceding claims, wherein the method further comprises the following steps: - defining a first set of line segments (33) by successive points of said first set of 3D positions (5), - defining a second set of line segments (34) through consecutive points of said second set of 3D positions (6), and - determining a transformation to said second set of line segments (34) such that said second set of line segments (34) best fits said first set of line segments (33).

6. The method according to any one of the preceding claims, wherein the method further comprises the following steps: - dividing said second time window (T2) into sub-windows, wherein in each said sub-window a subset of 3D positions is identified, and - for each of the 3D position subsets, calculating a corresponding transformed 3D position set, wherein each of the transformed 3D position sets is obtained by transforming the second 3D position set in the sub-window such that each transformed 3D position set best fits the first 3D position set.

7. The method according to any one of the preceding claims, wherein the method further comprises the following steps: - based on the second set of 3D positions (6) and the first set of 3D positions (5), inferring a predicted set of 3D positions (18) for an upcoming time instance (T3), wherein the inferring comprises identifying a trajectory (19) of the object (3) between the first time window (T1) and the second time window (T2), - identify a set of real 3D locations, and - Validating the predicted set by comparing it with the true set.

8. The method according to any one of the preceding claims, wherein the method further comprises the steps of: The first time window and the second time window (T1, T2) and the points therein are combined.

9. An optical sensor (21) for simultaneous localization and mapping (SLAM), comprising: - a first plurality (28) of pixel sensors (22), each pixel sensor (22) comprising a photodetector (23) and - a processing unit (24), wherein the photodetector (23) of the first plurality (28) of pixel sensors is adapted to output a second set (2) of locations of points of at least one object (3) in a scene (4) relative to the first plurality (28) of pixel sensors within a second time window (T2), wherein the second set (2) of locations is in 2D, wherein the unit (24) is adapted to convert the second set (2) of locations into a second set (6) of 3D positions, The method is characterized in that the unit (24) comprises a processor, which in use determines a transformation to the second set of 3D positions (6) such that the second set of 3D positions (6) best fits the first set of 3D positions (5), wherein the first set of 3D positions (5) corresponds to a first set of locations (1), wherein the first set of locations (1) is in 2D, wherein the first set of 3D positions (5) is denser than the second set of 3D positions (6), and wherein the first set of 3D positions (5) is within a first time window (T1) before the second time window (T2), and wherein the second time window (T2) is shorter than the first time window (T1), preferably at least twice shorter, The unit (24) is adapted to obtain each of the first set of 3D positions (5) and the second set of 3D positions (6) by triangulating two sets of locations in 2D relative to two different reference points (28, 29).

10. An optical sensor (21) according to claim 9, wherein the photodetector (23) of the first plurality (28) of pixel sensors is further adapted to output the first set of locations (1) of points of the at least one object (3) in the scene (4) within the first time window (T1), wherein the unit is further adapted to convert the first set of locations (1) into the first set of 3D positions (5).

11. The optical sensor (21) according to any one of claims 9 to 10, wherein the sensor (21) further comprises primary optics capable of producing an image of the scene (4) on the first plurality (28) of pixel sensors (22).

12. An optical sensing device (26), comprising an optical sensor (21) according to any one of claims 9 to 11, wherein the device (26) further comprises at least one light source (27), wherein the light source (27) is adapted to illuminate at least one point-like object (8) on the scene (4) along an illumination trace (9), wherein the device (26) further comprises a second plurality (29) of pixel sensors (22), each pixel sensor (22) comprising a photodetector (23), wherein the photodetectors (23) of the first plurality (28) of pixel sensors are adapted to monitor the point-like object (8) and output the first set of locations and the second set of locations (1, 2) along the trace (9), and wherein the second plurality (29) of pixel sensors The photodetector (23) of the sensor is adapted to monitor the point-like object (8) and output a third set of locations (11) of the points of the object (3) along the track (9) within the first time window (T1) and a fourth set of locations (11, 12) of the points of the object (3) along the track (9) within the second time window (T2) relative to the second plurality (28) of pixel sensors, wherein the unit (24) is capable of triangulating the first set of locations and the third set of locations (1, 11) to obtain the first 3D position set (5), and wherein the unit (24) is capable of triangulating the second set of locations and the fourth set of locations (2, 12) to obtain the second set of 3D positions (6).

13. The optical sensing device (26) according to claim 12, wherein the device (26) further comprises at least one memory element (25), wherein the element (25) is adapted to store at least the first set of 3D positions (5) and / or the second set of 3D positions (6).

14. An optical sensing device (26) according to any one of claims 12 to 13, wherein the unit (24) is capable of inferring a predicted 3D position set (18) of an upcoming instance (T3) based on the second 3D position set (6) and the first 3D position set (5), wherein the unit (24) is capable of identifying a trajectory (19) of the object (3) between the first time window (T1) and the second time window (T2), wherein the unit (24) is also capable of verifying the predicted set (18) by comparing it with a real 3D position set.

15. An optical sensing system comprising an optical sensing device (26) according to claims 12 to 14, wherein the system further comprises auxiliary optics capable of producing an image of the scene (4) on the second plurality (29) of pixel sensors (22).