3D Reconstruction Method

Abstract

The present invention relates to a method for 3D reconstruction of a scene (2). The method includes the step of providing a first camera (3) having a known position and orientation. Furthermore, the method includes the step of determining a first set of projection lines corresponding to pixels (11) of the first camera (3), the projection lines being defined by the position and orientation of the camera (3) and the corresponding pixels (11). Furthermore, the method includes the step of projecting a light beam (13) onto the scene (2) using a light source (12) to form a dot (1) on the scene (2). Furthermore, the method includes the step of determining pixels corresponding to the image of the dot (1) on the first camera (3), thereby defining a corresponding first projection line (9) from the first set of projection lines. Furthermore, the method includes the step of determining a second projection line (10) from a second set of projection lines corresponding to the instantaneous position and orientation of the light beam (13) or to the image of the dot (1) on a second camera (4). Furthermore, the method includes the step of calculating the 3D position (17) of the dot (1) on the scene (2) based on the first and second projection lines (9,10), wherein the 3D position (17) is defined in the region (18) where the first projection line (9) and the second projection line (10) are closest to each other.

Description

Technical Field

[0001] The present invention relates to a 3D reconstruction method.

Background Art

[0002] There are several methods for reconstructing a three-dimensional scene, one of which is triangulation. In this method, two cameras are used to capture scene information, and the two-dimensional data of the two cameras is triangulated. First, the two-dimensional data of the scene is acquired by each camera, and then the two-dimensional data of both cameras is triangulated to obtain the three-dimensional information of the scene. This is achieved by performing the triangulation using mathematical operations for each point in the scene. However, this process requires time and energy.

[0003] Therefore, there is a need for a method that enables 3D reconstruction with low power consumption and low latency. The present invention aims to solve the above problems at least in part.

Summary of the Invention

[0004] An object of an embodiment of the present invention is to provide a fast and low-power 3D reconstruction method for 3D reconstruction. The above object is achieved by the system and method according to the present invention

[0005] A first aspect of the present invention relates to a method for 3D reconstruction of a scene and includes the following steps. <000...]] - Supplying a first camera having a known position and orientation. - Determining a first set of projection lines corresponding to the pixels of the first camera. The projection lines are defined by the position and orientation of the first camera and the corresponding pixels. - Projecting a light beam onto the scene by a light source, thereby forming dots on the scene. <000...]] - A step of determining the pixels corresponding to the image of the dot on the first camera, thereby defining the corresponding first projection line from the first set of projection lines. - A step of determining a second projection line from a second set of projection lines corresponding to the instantaneous position and orientation of the light beam, or to the image of the dot on the second camera. - A step of calculating the 3D position of the dot on the scene based on the first and second projection lines. The 3D position is defined in the region where the first projection line and the second projection line are closest to each other.

[0006] According to embodiments of the present invention, since the 3D position is acquired based on predefined, for example, pre-calculated or pre-set projection lines, there is an advantage that high-speed 3D reconstruction is possible. In addition, since the amount of processing required is small, there is an advantage that power consumption is reduced. Furthermore, since this method only requires the storage of projection lines, according to embodiments of the present invention, there is an advantage that the amount of memory required is small. Because the projection lines are pre-calculated, according to embodiments of the present invention, even when using more than two optical sensors, the required processing time and memory amount do not scale, which is an advantage.

[0007] A preferred embodiment of the first aspect of the present invention includes one of the following features, or a suitable combination of two or more of them.

[0008] The region is preferably defined on a line perpendicular to the first projection line and the second projection line, and preferably the 3D position is located midway between the first and second projection lines. Embodiments of the present invention have the advantage that, by such assumptions, the 3D position can be easily and quickly determined, and at the same time, a good estimate of the 3D position can be obtained.

[0009] If no predefined projection lines exist in the first and / or second projection line sets, the method preferably further includes the steps of defining the first projection line on the first projection line set and / or defining the second projection line on the second projection line set. Embodiments of the present invention have the advantage that the sets (e.g., lookup tables) can be refilled even if system changes such as calibration occur.

[0010] Preferably, the method further includes the step of updating the projection lines in the first and / or second projection line set. Embodiments of the present invention have the advantage that the projection lines can be updated when recalibration is performed.

[0011] The method preferably includes the step of scanning the light beam along a trajectory on the scene, wherein the trajectory includes a plurality of points on the scene. Embodiments of the present invention have the advantage that stereo matching is not required because the processing is performed based on the position of the light spot.

[0012] The first and / or second projection line sets are preferably sequentially filled with projection lines. Embodiments of the present invention have the advantage that a lookup table can be used when the necessary information is available, and if the necessary information is not available, the lookup table can be refilled and used in the future.

[0013] The method preferably includes the step of arranging multiple sensing units in M ​​rows and N columns, where each projection line or a predetermined number of projection lines corresponds to at least one pixel. Embodiments of the present invention have the advantage of reducing the amount of memory required to store the lookup table. Embodiments of the present invention have the advantage of being able to adjust the memory for storing the projection lines. Embodiments of the present invention have the advantage of being able to adjust the accuracy of the projection lines.

[0014] This method preferably includes the step of providing a single-photon detector, preferably a SPAD, within each pixel. Embodiments of the present invention have the advantage of being able to quickly detect the position of the light spot due to the high speed of the detector.

[0015] A second aspect of the present invention relates to an optical sensing system for 3D reconstruction of a scene, and includes the following: - A first camera having multiple pixels, each pixel including a photodetector. The first camera has a known position and orientation. - A light source adapted to project a light beam onto a scene, thereby forming dots on the scene. - An optical system capable of generating an image of the scene on the first camera. - A memory element including a first and second projection line set. The first set of projection lines corresponds to the pixels of the first camera. The projection lines are defined by the position and orientation of the camera, and the corresponding pixels. This system is adapted to determine the pixels corresponding to the image of the dot on the first camera, thereby defining the corresponding first projection line from the first set of projection lines. The system is further adapted to determine a second projection line from a second set of projection lines corresponding to the instantaneous position and orientation of the light beam, or to the image of the dot on a second camera having a known position and orientation. The system is adapted to calculate the 3D position of the dot on the scene based on the first and second projection lines, the 3D position being defined in the region where the first and second projection lines are closest to each other.

[0016] A preferred embodiment of a second aspect of the present invention includes one of the following features, or a suitable combination of two or more of them. - The region is defined on a line perpendicular to the first projection line and the second projection line, and preferably the 3D position is located midway between the first and second projection lines. - This system is adapted to define a first projection line on the first projection line set and a second projection line on the second projection line set if no predefined projection lines exist in the first and / or second projection line set. - The system further includes scanning means, which are adapted to scan the light beam along a trajectory on the scene, the trajectory including a plurality of points on the scene. - This system is adapted to sequentially fill the memory elements with projection lines. - The pixels are arranged in M ​​rows and N columns, and each projection line or a predetermined number of projection lines corresponds to at least one pixel. - The photodetector is a single-photon detector, preferably a SPAD.

[0017] The above and other features, characteristics and advantages of the present invention will become apparent from the following detailed description, which is accompanied by accompanying drawings illustrating the principles of the present invention. This description is provided for illustrative purposes only and does not limit the scope of the present invention. [Brief explanation of the drawing]

[0018] This disclosure is further described by the following description and accompanying drawings. [Figure 1] Figure 1 shows a system (100) for 3D imaging according to an embodiment of the present invention. [Figure 2] Figure 2 is a diagram relating to an embodiment of the present invention, showing the 3D position (17) of the light spot (1) from two different viewpoints. [Figure 3] Figure 3 shows an embodiment of the present invention, illustrating (a) two projection lines (9,10) corresponding to two optical sensors (e.g., a camera) and (b) three projection lines (9,10,19) corresponding to three optical sensors. [Figure 4]FIG. 4 is a diagram showing how 3D positions (31 - 34) are defined by projection lines (20 - 23) and (29) corresponding to different sensing units (e.g., pixels) (24 - 27) and (30) with respect to an embodiment of the present invention. [Figure 5] FIG. 5 is a diagram showing a configuration of a system (100) similar to that of FIG. 1 but having only one optical sensor (3) with respect to an embodiment of the present invention. The reference numerals attached to the claims should not be construed as limiting the scope of the present invention. In different drawings, the same reference numerals indicate the same or corresponding elements.

Embodiments for Carrying Out the Invention

[0019] The present invention relates to a method for 3D reconstruction of a scene and a system therefor.

[0020] The present invention will be described with reference to specific embodiments and specific drawings, but the present invention is not limited thereto and is limited only by the claims. The described drawings are schematic and not limiting. In the drawings, for ease of explanation, the sizes of some elements may be exaggerated and are not necessarily drawn to scale. Dimensions and relative dimensions do not correspond to the scale in actual implementation of the present invention.

[0021] Terms such as first, second, etc. in this specification and the claims are used to distinguish similar elements and are not necessarily used to describe an order in terms of time, space, rank, or any other way. When used in this way, the terms are replaceable under appropriate circumstances, and it should be understood that the embodiments of the present invention described in this specification can operate in an order other than that described or illustrated in this specification.

[0022] Terms such as “top” and “under” used herein and in the claims are for illustrative purposes only and are not necessarily used to describe relative positions. These terms are interchangeable under appropriate circumstances, and it should be understood that embodiments of the present invention described herein may operate in orientations different from those described or illustrated herein.

[0023] Numerous specific details are provided in the description herein. However, it will be understood that embodiments of the present invention can be carried out without these specific details. In other examples, well-known methods, structures, and techniques are not described in detail so as not to hinder the understanding of this description.

[0024] Similarly, in the description of exemplary embodiments of the Invention, it should be understood that various features of the Invention may be summarized in a single embodiment, drawing, or description for the purpose of streamlining the disclosure and aiding in the understanding of one or more different aspects of the Invention. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed Invention requires features beyond those explicitly stated in each claim. Rather, as reflected in the following claims, aspects of the Invention exist in fewer features than all the features of the single embodiment disclosed previously. Accordingly, the claims following the detailed description are explicitly incorporated into the detailed description herein, and each claim stands on its own as a separate embodiment of the Invention.

[0025] Furthermore, while some embodiments described herein include some features included in other embodiments, others do not. However, as those skilled in the art will understand, combinations of features from different embodiments fall within the scope of the invention and may form different embodiments. For example, any of the claimed embodiments may be used in any combination within the following claims.

[0026] Unless otherwise defined, all terms used in disclosing this invention, including technical and scientific terms, have the meanings generally understood by those skilled in the art to which this invention pertains. For further guidance, definitions of terms are included to better understand the teachings of this invention.

[0027] In this specification, the following terms have the meanings set forth below.

[0028] As used herein, "A," "an," and "the" refer to both singular and plural nouns unless explicitly stated otherwise in the context. For example, "a contaminant" refers to one or more contaminants.

[0029] When specifying a numerical range using endpoints, the range includes not only the specified endpoints but also all numerical values ​​and fractions contained within that range.

[0030] In a first embodiment, the present invention relates to a method for 3D reconstruction (e.g., depth estimation) of a scene. The method includes the step of supplying a first optical sensor (e.g., a first camera) having a known position and orientation. The method further includes the step of determining a first set of projection lines corresponding to sensing units (e.g., pixels) of the first optical sensor. The projection lines are defined by the position and orientation of the first optical sensor and the corresponding sensing units on the optical sensor. For example, the plurality of sensing units are arranged in a matrix configuration. The method further includes the step of projecting a light beam onto the scene by a light source, thereby forming a light spot (e.g., a dot) on the scene. For example, the light spot may be a dot, a half-dot, a collection of multiple dots, or any other suitable shape.

[0031] The method further includes the step of determining a pixel corresponding to the image of a light spot on the first optical sensor, thereby defining a corresponding first projection line from the first set of projection lines. For example, the detection location on the optical sensor (i.e., which sensing unit detected the light spot) and the corresponding projection line are determined.

[0032] The method further includes the step of determining a second projection line from a second set of projection lines corresponding to the instantaneous position and orientation of the light beam, or to the image of the light spot on a second optical sensor (e.g., a second camera).

[0033] Preferably, the method includes the step of imaging the scene onto the optical sensor or group of optical sensors, for example, using a suitable optical system (e.g., at least one imaging optical system) capable of imaging the scene onto the optical sensor or group of optical sensors. For example, light reflected from the scene is received by at least one sensing unit of the plurality of sensing units of the optical sensor or group of optical sensors. The light is detected by the group of optical sensors, where the light pulse is converted into an electrical signal or logic pulse. Each pulse is timestamped with respect to a reference clock. The detected signal is then filtered using, for example, time information and proximity information.

[0034] Advantageously, a first position (i.e., detection position) of the image of the light spot on the first optical sensor is determined, and a second position is also determined. The second position is the instantaneous position and orientation of the light beam of the light source. For example, if the light source is considered to be a camera or optical sensor, the image plane corresponding to the light source will include dots that form a scanning pattern (e.g., a Lissajous pattern) instead of viewing the light spot with the sensing unit.

[0035] For example, projection lines are known. For instance, the system includes scanning means (as shown below) for scanning a light beam from a light source across a scene, and projection lines are known for each angle scanned by the scanning means.

[0036] Alternatively, the second position is the position of the image of the light spot on the second optical sensor. For example, the first and second positions are positions on the sensing unit, for example, the position of the sensing unit on the optical sensor. For example, this step determines on which sensing unit the light spot is imaged. Since the position and orientation of the camera are known, it is possible to define the position and orientation of each pixel.

[0037] Preferably, the method associates the first position with a first projection line predefined in a first lookup table, and the second position with a second projection line predefined in a second lookup table. For example, the lookup tables are stored in memory. For example, the lookup tables are arranged in a matrix configuration and correspond to, for example, sensing units. For example, each sensing unit corresponds to one projection line.

[0038] The first optical sensor is defined by its position and orientation in a common global coordinate system, i.e., its position and orientation are represented in three dimensions in the common global coordinate system. Similarly, the light source or second optical sensor is also defined by its position and orientation in a common global coordinate system (e.g., a different position from the first optical sensor, i.e., the same or different orientation as the first optical sensor), i.e., its position and orientation are represented in the common global coordinate system. Orientation can be defined, for example, by indicating how the sensor is pointed, i.e., in which direction it is facing, for example, in degrees, for example, relative to the common global coordinate system. Orientation may also be represented, for example, as a vector passing through the camera position and the detected position on the camera.

[0039] The global coordinate system is common to the second optical sensor, the light spot, the projection lines, and so on. In other words, the global coordinate system is common to all components of this system.

[0040] If the system includes one optical sensor and one light source, then at each moment, the location of the light source, for example, the instantaneous position and orientation of the light beam, is known. Therefore, advantageously, the projection line can be defined based on the light source instead of the second optical sensor. For example, the second position and the second projection line can be defined based on the light source, as in the case of the second optical sensor. However, in this case, the second position must be estimated based on, for example, the trajectory (the scanning trajectory of light on the scene, e.g., a Lissajous pattern) and the scanning frequency of the scanning means (e.g., MEMS), and the scanning frequency is defined based on the time it takes for the light to scan the scene from one end to the other.

[0041] The method further includes the step of calculating the 3D position of the light spot on the scene. For example, the position relative to a first optical sensor, or the position relative to the first and second optical sensors. The 3D position is defined in the region where the first projection line and the second projection line are closest to each other. Those skilled in the art will understand that the two projection lines do not actually intersect but reach the point where they are closest to each other in the region. This is due to measurement errors such as quantization errors. For example, it is assumed that the 3D position of the light spot is located midway between the first and second projection lines in the region where the first and second projection lines are closest to each other. If there are three optical sensors, and therefore three projection lines, the 3D position is calculated in the region where all three projection lines are closest to each other. This will be further illustrated in the figures below. Those skilled in the art will understand how to calculate the distance between lines and how to find the position of the minimum distance between two lines.

[0042] Because the 3D position is based on pre-calculated projection lines, it is not necessary to calculate the projection lines every time a 3D position calculation is required. This saves a significant amount of processing time and therefore improves the processing speed of 3D reconstruction. As a result, power consumption is also reduced.

[0043] Another advantage is that less memory is required to store the projection lines. As mentioned above, each projection line is defined by the detection position and orientation of the optical sensor, for example, by a ray or vector. Therefore, the amount of memory required is small.

[0044] Another important advantage is that, when there are more than two optical sensors, the processing time and memory required when projection lines are pre-calculated do not scale as rapidly as when they are not. When pre-calculated, the required memory and time are proportional to the number of sensors (N), but when not pre-calculated, they are exponentially proportional to the number of sensors (N factorial). For example, if there are three optical sensors, the projection lines corresponding to each optical sensor must be calculated twice. For example, the first 3D position is calculated by calculating the first and second projection lines, then the second 3D position is calculated by calculating the first projection line (second time) and the third projection line, and then the third 3D position is calculated by calculating the second projection line (second time) and the third projection line (second time). The final 3D position may be the average of the first, second, and third 3D positions. Alternatively, the first, second, and third 3D positions may have different weights depending on the error measurement criteria. For example, if a 3D position corresponds to two optical sensors that are close together, the weight of that 3D position will be smaller than the weight of a 3D position corresponding to two optical sensors that are farther apart. This is because the further apart the two optical sensors are, the longer the baseline becomes, and the more accurate the 3D position becomes. Alternatively, the 3D position may be the point that minimizes the distance (or weighted distance) to all projection lines. Many other implementations are also possible.

[0045] The reason why the projection lines need to be calculated twice is that these projection lines are not stored but are calculated on the fly, and the 3D position is calculated based on that calculation. This shows that the time and energy required scale dramatically when there are more optical sensors, especially when there are more than two or three optical sensors.

[0046] The trade-off between memory and time should be kept in mind; that is, as processing speed decreases, the amount of memory required increases.

[0047] Those skilled in the art will understand that, for example, instead of knowing the precise positions and orientations of the first camera, the second camera, and the light source, it may be sufficient to know, for example, the relative positions and orientations between the first camera and the light source, or between the first camera and the second camera.

[0048] Preferably, the region where the first and second projection lines are closest to each other is defined on or near a line perpendicular to the first and second projection lines, where the perpendicular line is, for example, perpendicular to and passes through both the first and second projection lines. For example, the intersection of the perpendicular line and the projection lines is found, and the perpendicular line is defined based on the intersection. Preferably, the 3D position is located midway between the first and second projection lines. In other words, assuming that the projection lines are vectors pointing to the scene, the perpendicular line is the cross product of these vectors, and the 3D position is the midpoint of the perpendicular line and the midpoint of the first and second projection lines. This is an assumption that can be done easily and quickly and, in most cases, has good accuracy.

[0049] The term "midway" may have a tolerance of 10% or 20%. That is, the 3D position may not be exactly midway between the projection lines, but rather 10% or 20% closer to either projection line.

[0050] Preferably, if the system includes three or more optical sensors and thus three vertical lines are obtained, each of the vertical lines is assigned a weighted average based on the length of the vertical line. For example, the length of the vertical line may be set to be inversely proportional to the weight; that is, the shorter the vertical line, the greater its weight. This is because the shorter the vertical line corresponding to two projection lines, the more likely the 3D point is to be close to the projection line.

[0051] Preferably, if no predefined projection lines exist in the first and / or second set of projection lines, the method further includes the steps of defining the first projection line on the first lookup table and / or defining (e.g., filling) the second projection line on the second lookup table. For example, this applies when no projection lines associated with a detected position exist. For example, the lookup table may be pre-calculated in a preliminary stage before use. The lookup table may also be empty in its initial state. In this case, the table must first be filled with projection lines. For example, the projection line corresponding to each sensing unit only needs to be calculated once, and thereafter it is possible to calculate the 3D position using that projection line. In practice, calibration is required from time to time. For example, if something changes to the system, recalibration may be required. In this case, the projection lines should be recalculated because they are valid for only one calibration.

[0052] Preferably, the method includes the steps of retrieving a previously stored projection line from the lookup table that corresponds to any of the plurality of points, and calculating the 3D position of the point based on the projection line.

[0053] Preferably, the method further includes the step of updating the projection lines in the first and / or second projection line sets, for example, if the calibration is off, i.e., if the system needs to be recalibrated.

[0054] Preferably, the method includes the step of scanning the light beam along a trajectory on the scene, the trajectory including a plurality of points on the scene. Since the method processes based on the position of the light spot, stereo matching is not required. For example, images on two optical sensors can be matched using the light spot.

[0055] The 3D position is calculated for each point along the trajectory. This is done based on predefined projection lines (as described above), or, if no projection lines exist, by first defining the projection lines and then using those projection lines when the same light spot is scanned a second or third time. The scanning is preferably performed continuously so that objects in the scene are continuously scanned and identified. For example, a light source generates a light beam to form a light spot on an object, and the light beam continuously scans the scene along the trajectory. For example, at regular intervals, the light beam scans the entire or nearly entire scene. The reflected signal is received by an optical sensor. The scanning may be performed by, for example, a Lissajous pattern, raster scanning, etc. The scanning means may be, for example, a MEMS scanner, mirrors, optical phase arrays, or beam scanning by a metasurface approach. By scanning the scene, it becomes possible to calculate the 3D position in different parts of the scene.

[0056] Preferably, the first and second lookup tables (i.e., projection line sets) are sequentially filled with projection lines. For example, each projection line corresponds to a single point in the scene. For example, if a projection line exists in the lookup table, it can be used. However, if it does not exist due to a change in the system's calibration, the lookup table can be refilled and used in the future. The idea of ​​sequentially filling projection lines is advantageous in that, after each calibration, it is not necessary to wait to find and save all projection lines before starting 3D reconstruction. Instead, the projection lines are saved sequentially, for example, in stages, during the 3D reconstruction process until all are refilled or the calibration is changed. For example, during the execution of 3D reconstruction, projection lines corresponding to different 3D locations are saved sequentially.

[0057] Preferably, the method includes the step of arranging a plurality of sensing units in M ​​rows and N columns, for example, each projection line or a predetermined number of projection lines corresponds to at least one sensing unit. If each projection line corresponds to one sensing unit, the number of projection lines per optical sensor is M × N. If each projection line is represented by three floating-point numbers (xyz) indicating position and three floating-point numbers (xyz) indicating direction, and each of these floating-point numbers is encoded with 32 bits, the memory size required to store the projection lines is 192 × M × N bits per optical sensor.

[0058] Alternatively, each projection line may correspond to two sensing units, for example, the sensing units may be divided into groups of two. For example, the projection line is the average of the projection lines of the two groups of sensing units. That is, the projection lines of both sensing units are found, then the average projection line is found, and the average value is stored in memory. In this case, the number of projection lines is reduced to 0.5 × M × N, i.e., 96 × M × N bits. This configuration has the advantage of reducing memory consumption, for example, by reducing the amount of data in the lookup table.

[0059] Alternatively, each sensing unit may correspond to two projection lines. For example, projection lines can be calculated based on the assumption that each sensing unit is computationally divided into two sensing units. In this case, the number of projection lines per optical sensor doubles to 2 × M × N, i.e., 384 × M × N bits. This configuration is advantageous in improving the accuracy of 3D positioning. Different implementations are also conceivable, such as the assumption that each sensing unit is divided in two dimensions (i.e., divided into four sensing units).

[0060] A preferred implementation from the above-mentioned options can be selected depending on the situation and can be readjusted, for example, according to the required amount of data to be stored in the lookup table and the required accuracy of the projection lines.

[0061] Preferably, the method includes the step of providing a single-photon detector within each sensing unit. This has the advantage that the position of the light spot can be detected quickly and accurately because such a detector is inherently fast. The single-photon detector can also minimize the energy required to detect the light spot. In conventional systems that do not use a single-photon detector, detection is slow, and for example, the scanned light spot appears not as individual light spots on the scene, but as a linear or complex structured light pattern, in which case stereo matching is required. However, by detecting the scanned light spot (e.g., a dot) using a single-photon detector, the need for stereo matching is eliminated.

[0062] In a second embodiment, the present invention relates to a system for 3D reconstruction of a scene. The system includes at least a first optical sensor, or at least first and second optical sensors, wherein the optical sensor or group of optical sensors (i.e., a camera) has a plurality of sensing units (e.g., in a matrix configuration), each of which includes a photodetector. Each of the optical sensor or group of optical sensors has a known position and orientation. Furthermore, the system includes at least one light source, which projects a light beam onto the scene, thereby being adapted to form, for example, dots on the scene. Preferably, the system further includes an optical system capable of generating an image of the scene on the optical sensor. For example, light strikes an object in the scene and is then reflected by the photosensor. Furthermore, the system further includes a memory element. For example, the memory element includes two lookup tables (i.e., first and second projection line sets) corresponding to each optical sensor.

[0063] The first set of projection lines corresponds to the pixels of the first camera. The projection lines are defined by the position and orientation of the camera, and the corresponding pixels.

[0064] This system is adapted to determine the pixels corresponding to the image of a dot on the first camera, thereby defining the corresponding first projection line from the first set of projection lines.

[0065] The system has been further adapted to determine a second projection line from a second set of projection lines corresponding to the instantaneous position and orientation of the light beam, or to the image of a dot on a second camera.

[0066] Preferably, the system is adapted to read a first position of the image of the light spot on the first optical sensor, for example, the detection position of the light spot on the optical sensor, and further read a second position. The second position may be the instantaneous position and orientation of the light beam, or it may be the position of the image of the light spot on the second optical sensor.

[0067] For example, if the system has first and second optical sensors, the first and second positions correspond to a sensing unit. That is, for example, the positions of light spots imaged on the first and second optical sensors correspond to the positions of the sensing unit, i.e., the positions of the sensing unit that receives the image of the light spots.

[0068] Preferably, the system is adapted to associate a first projection line, predefined in the memory element (for example, one of the lookup tables), with a detection position on the first optical sensor.

[0069] For example, the system is adapted to associate a predefined second projection line in the memory element (e.g., the other of the lookup table) with a detection position on the second optical sensor.

[0070] This system is adapted to calculate the 3D position of the dot on the scene based on the first and second projection lines, and the 3D position is defined in the region where the first and second projection lines are closest to each other.

[0071] Preferably, the region where the first and second projection lines are closest to each other is defined on lines perpendicular to the first and second projection lines. For example, the 3D position is located midway between the first and second projection lines.

[0072] Preferably, in the case of predefined projection lines in the first and / or second projection line set, the system is adapted to define (e.g., fill) the first and second projection lines on the first and second projection line set.

[0073] Preferably, the system further includes scanning means, which are adapted to scan the light beam along a trajectory over the scene, the trajectory comprising a plurality of points on the scene. The scanning of the scene is performed at least partially, preferably over a large portion (or all) of the scene for each of the plurality of points. For example, the very first light spot also constitutes part of the plurality of points. This is advantageous for filling a lookup table, as projection lines corresponding to many or all of the points constituting the scene can be found.

[0074] Scanning the light beam from the light source is advantageous in that it enables triangulation. For example, by using a system having a light source that irradiates a scene (e.g., environment) with a light beam and two optical sensors that share a field of view of the scene, it is possible to convert the xy-time data of the two optical sensors at each timestamp into xyz-time data by triangulation. For example, if the light source is adapted to irradiate the scene with a light beam along an irradiation trajectory, the light source includes means for scanning the light beam over the scene (preferably continuously), the two optical sensors monitor the light spot generated by the light beam and output the position of at least one object in the scene (e.g., a point on the surface of the object) along the trajectory over multiple time (instances), and the xy-time data of the two optical sensors can be converted into xyz-time data using triangulation. The light source, for example, functions as a reference point, and this reference point can be used to triangulate the position of the point on the object as measured by the first optical sensor and its position as measured by the second optical sensor to generate depth or the z-dimension. However, in the context of the present invention, it is preferable to determine the 3D position by defining projection lines and the region where these projection lines are closest to each other, as described above, and by defining vertical lines and midpoints on the vertical lines.

[0075] Preferably, the system is adapted to sequentially fill the memory elements with projection lines, for example, the first and second projection lines. For example, as scanning progresses, more projection lines are stored in the memory elements.

[0076] Preferably, the sensing units are arranged in M ​​rows and N columns, with each projection line or a predetermined number of projection lines corresponding to at least one sensing unit.

[0077] Preferably, the photodetector is a single-photon detector, and more preferably a single-photon avalanche diode (SPAD). Alternatively, the photodetector may be an avalanche photodetector. Using a SPAD within a pixel (i.e., a sensing unit) has the advantage that, because the detector is sensitive to single photons, the energy required to detect the actively projected structure can be minimized. Furthermore, since SPADs have a response time of less than nanoseconds, they have the advantage that photons are detected and converted into digital signals in nanoseconds.

[0078] Preferably, a filtering step may be performed prior to triangulation, or in this case, calculation of the 3D position. For example, this is to reduce the number of sensing units that have produced false detections, while triangulating only the data from sensing units that have obtained true detections. For example, detections of adjacent sensing units (e.g., pixels) or the persistence of detections over time may be checked. For example, these filters are described in PCT IB2021 054688, PCT EP2021 087594, PCT IB2022 000323, and PCT IB2022 058609. This is an in-pixel filtering mechanism. Other filtering methods are also available; for example, after projecting the detection results onto a surrounding area, noisy detections can be removed because the projection pattern (e.g., Lissajous pattern) is known.

[0079] For example, filtering may be performed based on the expected pulse width, the expected dot projection size, and past and current detection results for pixels and their adjacent pixels. The detection signal is then output to row and column signal buses, which makes it possible to determine the position (i.e., xy coordinates) of the laser dot at each timestamp. For example, filtering may be performed based on past bus information and used to identify the detection location that is most likely to be detecting photons originating from active irradiation.

[0080] Preferably, each photodetector is arranged in a reverse-bias configuration. Preferably, the detector is capable of detecting a single photon incident thereon. Preferably, the photodetector is adapted to output a logic signal, such as an electrical detection signal, when a photon is detected. For example, the detection signal is represented by a signal including logic "1", e.g., detected, while no signal detected is represented by a signal including logic "0", e.g., no detection. Alternatively, the detection signal may be represented by a pulsed signal, such as a transition from logic "0" to logic "1" and then back from logic "1" to logic "0", or may result from such a pulsed signal. On the other hand, no detection may be represented by the absence of such a pulsed signal, or may result from it.

[0081] Preferably, the light source is matched to a wavelength detectable by the sensing unit of the optical sensor. For example, between 100 nanometers and 10 micrometers, preferably between 100 nanometers and 1 micrometer. For example, the optical sensor may be a photodetector or a matrix of photodetectors capable of detecting photons incident on each detector within a wavelength detection window located in the range of 100 nanometers to 10 micrometers, preferably between 100 nanometers and 1 micrometer.

[0082] Preferably, the system comprises 100 or more pixel sensors (i.e., sensing units), preferably 1,000 or more pixel sensors, more preferably 10,000 or more pixel sensors, even more preferably 100,000 or more pixel sensors, and most preferably 1,000,000 or more pixel sensors. For example, the system may be arranged in a matrix configuration including rows of 1,000 pixel sensors and columns of 1,000 pixel sensors.

[0083] Preferably, the system includes image display means, such as a screen or other image display device, to reproduce the position of the points of the objects in the scene.

[0084] Preferably, the optical system is used for 3D vision applications. For example, the system is used to visualize objects in three dimensions. Alternatively, the sensor can enable analysis of a scene, for example, by extracting features of objects in the scene, without necessarily generating an image of the scene. Preferably, the system further comprises multiple optical sensors and / or multiple light sources. This is advantageous for creating a 3D vision by, for example, triangulating the outputs (i.e., xy data) of two optical sensors to obtain 3D information (xyz data).

[0085] Any feature of the second aspect (system) is as described in the first aspect (method) as its corresponding feature.

[0086] In a third embodiment, the present invention relates to 3D reconstruction of a scene using the method according to the first embodiment, and / or 3D reconstruction of a scene using the system according to the second embodiment.

[0087] Further features and advantages of embodiments of the present invention will be described with reference to the drawings. It should be noted that the present invention is not limited to the specific embodiments shown in these drawings or described in the examples, but is limited only by the claims.

[0088] Figure 1 shows a system (100) for 3D imaging according to an embodiment of the present invention. The system (100) includes a first optical sensor (3) and a second optical sensor (4). Each optical sensor (3,4) includes a plurality of sensing units (11). The system further includes a light source (12) which projects a light spot (1) generated by a light beam (13) onto a scene (2). The light beam (13) is scanned along a trajectory (16) on the scene (2), the trajectory (16) includes a plurality of points (14,15). These points include the point where the light spot (1) is first located.

[0089] The system (100) further includes an optical system capable of generating an image of the scene (2) on the optical sensors (3,4). For example, an image of a light spot (1) is generated on the sensing units (11) of the first and second optical sensors (3,4). For example, the position of the light spot (1) generated on the sensing unit (11) of the first optical sensor (3) is called the first position (5), and similarly, the second position (6).

[0090] Furthermore, for a single calibration of the optical sensors (3,4), first and second camera positions (7,8) (in 3D) are defined. The first and second camera positions (7,8) (in 3D) represent the global position of each optical sensor (3,4), as will be understood by those skilled in the art. For example, it is the global position in a global coordinate system common to the optical sensors (3,4), projection lines (9,10), and light spot (1). Those skilled in the art also understand that calibration is based on external factors, such as the position and / or orientation of the sensors (3,4), as well as internal factors (internal camera parameters such as focal length and principal point offset).

[0091] Figure 2 is a diagram relating to an embodiment of the present invention, showing the 3D position (17) of the light spot (1) from two different viewpoints: (a) a top view and (b) a side view. As shown in the figure, the 3D position (17) is located in the region (18) where the first projection line (9) and the second projection line (10) are closest to each other. Ideally, the point is located at the intersection of the two lines. However, in reality, the projection lines (9,10) do not intersect. Therefore, the 3D position is estimated to be located midway between the projection lines (9,10) by determining the average 3D position between the two projection lines (9,10), for example, as shown in (b). Those skilled in the art will understand how to determine the 3D position based on the known positions of detection in both optical sensors (cameras) and the external and internal parameters of the cameras. For example, use the information in this document: https: / / www.scratchapixel.com / lessons / 3d-basic-rendering / computing-pixel-coordinates-of-3d-point / mathematics-computing-2d-coordinates-of-3d-points.html

[0092] Figure 3 shows an embodiment of the present invention, illustrating (a) two projection lines (9,10) corresponding to two optical sensors and (b) three projection lines (9,10,19) corresponding to three optical sensors. In the first case, the 3D position (17) is calculated based on the first and second projection lines (9,10). Since the projection lines (9,10) are calculated in advance, it is only necessary to determine the 3D position based on the projection lines (9,10). If the projection lines (9,10) are not calculated in advance, it is necessary to calculate them first and then determine the 3D position.

[0093] The problem of not pre-calculating the projection lines (9,10) becomes clearer when there are three or more projection lines, for example, when there are first, second, and third projection lines (9,10,19) corresponding to three optical sensors, as shown in (b). In this case, since the projection lines are not usually saved, it is necessary to first calculate each projection line twice. For example, first calculate the first projection line (9) and the second projection line (10) to find the first 3D position, then calculate the first projection line (9) and the third projection line (19) to find the second 3D position, and then calculate the second projection line (10) and the third projection line (19) to find the third 3D position. Finally, the final 3D position is determined based on the first, second, and third 3D positions. This clearly increases processing time, meaning that the more optical sensors included in the system, the slower the processing speed becomes. Furthermore, the processing speed decreases even further when the resolution of the optical sensor, i.e., when the number of sensing units included in the optical sensor is large.

[0094] However, under the present invention, processing time does not scale so dramatically, meaning that processing speed does not decrease so dramatically. This is because the projection lines are pre-calculated, and therefore do not need to be calculated. Thus, as described above, it is only necessary to obtain the projection lines from a lookup table or memory, determine the three 3D positions, and then determine the final 3D position. Those skilled in the art understand that calculating the projection lines is a computationally intensive part of this type of 3D reconstruction. Those skilled in the art also understand that there is a trade-off between time and memory in the invention described above. For example, pre-calculation reduces processing time but increases memory consumption.

[0095] Figure 4 shows four different projection lines (20-23) corresponding to the four sensing units (24-27) of the second optical sensor (4) and one projection line (29) corresponding to one sensing unit (30) of the first optical sensor (3), along the epipolar line (28). These four projection lines (20-23) intersect with one projection line (29) at four different 3D positions (31-34). In other words, any 3D position in the scene can be defined by two projection lines. For example, the 3D position at position (30) is defined by projection lines (20) and (29), and similarly, the 3D position at position (33) is defined by projection lines (23) and (20). The same applies to other positions. By pre-calculating the projection lines, it becomes possible to define 3D positions in the scene based on these projection lines.

[0096] Figure 5 shows a system (100) for 3D imaging similar to that in Figure 1. However, Figure 5 only shows the first optical sensor (3). In this case, as described above, instead of the second optical sensor (4), the light source (12) and its scanning means are used to determine the second projection line (10).

[0097] Other configurations of methods and apparatus for achieving the objectives of the present invention will be obvious to those skilled in the art. The above description provides details relating to specific embodiments of the present invention. However, even though the above description is detailed, it is clear that the present invention can be carried out in a variety of forms. Furthermore, if specific terms are used in this specification to describe particular features or embodiments, those terms should not be construed as meaning that the present invention is intended to be limited to those features or embodiments. [Explanation of Symbols]

[0098] 1 Light spot 2 scenes 3. First optical sensor 4. Second optical sensor 5 1st position 6 2nd position 7. First camera position 8. Second camera position 9 1st projection line 10 Second projection line 11 Sensing Unit 12 light source 13 Light beam 14,15 Multiple points 16 Trajectory 17. 3D position of the light spot 18. Regions where projection lines are closest to each other. 19 Third projection line 20-23 Four projection lines 24-27 Four sensing units 28 Epipolar lines 29. One projection line 30 One sensing unit 31-34 Four different 3D positions 35. Perpendicular lines to projection lines 100 Systems

Claims

1. A 3D reconstruction method for 3D reconstruction of scene (2), The steps include supplying the first camera (3), The first camera (3) has a known relative position and orientation with respect to the light source (12) or the second camera (4), and the light source (12) projects a light beam (13) onto the scene (2), thereby forming a dot (1) on the scene (2), The steps include determining a first set of projection lines corresponding to the pixels (11) of the first camera (3), which is comprised of the known relative position and orientation of the first camera (3) and the projection lines defined by the corresponding pixels (11), The steps include determining the pixels corresponding to the image of the dot (1) on the first camera (3), thereby defining the corresponding first projection line (9) from the first set of projection lines, A step of determining a second projection line (10) from a second set of projection lines corresponding to the instantaneous position and orientation of the light beam (13), or to the image of the dot (1) on the second camera (4), The steps include: calculating the 3D position (17) of the dot (1) on the scene (2) based on the first and second projection lines (9, 10), and defining the 3D position (17) in the region (18) where the first projection line (9) and the second projection line (10) are closest to each other; Having, 3D reconstruction method.

2. In the 3D reconstruction method described in claim 1, The region (18) is defined on a line (35) perpendicular to the first projection line (9) and the second projection line (10), and preferably the 3D position (17) is located midway between the first and second projection lines (9, 10). 3D reconstruction method.

3. In the 3D reconstruction method according to either claim 1 or 2, If no predefined projection lines exist in the first and / or second set of projection lines, the 3D reconstruction method is as follows: The steps include defining the first projection line (9) on the set of first projection lines, and / or, The steps include defining the second projection line (10) on the set of second projection lines, Further including, 3D reconstruction method.

4. In the 3D reconstruction method according to any one of claims 1 to 3, The 3D reconstruction method further includes the step of updating the projection lines in the first and / or second set of projection lines. 3D reconstruction method.

5. In the 3D reconstruction method according to any one of claims 1 to 4, The 3D reconstruction method includes the step of scanning the light beam (13) along a trajectory (16) on the scene (2), wherein the trajectory (16) includes a plurality of points (14, 15) on the scene (2). 3D reconstruction method.

6. In the 3D reconstruction method according to any one of claims 1 to 5, The first and / or second set of projection lines is sequentially filled by projection lines. 3D reconstruction method.

7. In the 3D reconstruction method according to any one of claims 1 to 6, The 3D reconstruction method includes the step of arranging the plurality of sensing units (11) in M ​​rows and N columns, wherein each projection line (9, 10) or a predetermined number of projection lines (9, 10) corresponds to at least one pixel (11). 3D reconstruction method.

8. In the 3D reconstruction method according to any one of claims 1 to 7, The 3D reconstruction method includes the step of providing a single-photon detector, preferably a SPAD, within each pixel (11). 3D reconstruction method.

9. An optical sensing system (100) for 3D reconstruction of scene (2), A first camera (3) each includes multiple pixels (11) including a photodetector, A light source (12) is adapted to project a light beam onto the scene (2) thereby forming a dot (1) on the scene (2), wherein the first camera (3) has a known relative position and orientation to the light source (12) or the second camera (4), An optical system capable of generating an image of the scene (2) on the first camera (3) or the first and second cameras (3, 4), A memory element including a first and second projection line set, It has, The known relative position and orientation of the first camera (3), and the first set of projection lines defined by the corresponding pixels (11) correspond to the pixels (11) of the first camera (3), The optical sensing system (100) is adapted to determine the pixels corresponding to the image of the dot (1) on the first camera (3), thereby defining the corresponding first projection line (9) from the first set of projection lines. The optical sensing system (100) is further adapted to determine a second projection line (10) from a second set of projection lines corresponding to the instantaneous position and orientation of the light beam (13), or to the image of the dot (1) on the second camera (4), The optical sensing system (100) is adapted to calculate the 3D position (17) of the dot (1) on the scene (2) based on the first and second projection lines (9, 10), wherein the 3D position (17) is defined in the region (18) where the first projection line (9) and the second projection line (10) are closest to each other. Optical sensing system.

10. In the optical sensing system according to claim 9, The region (18) is defined on a line (35) perpendicular to the first projection line (9) and the second projection line (10), and preferably the 3D position (17) is located midway between the first and second projection lines (9, 10). Optical sensing system.

11. In the optical sensing system according to either claim 9 or claim 10, The optical sensing system (100) is adapted to define the first projection line (9) on the first projection line set and the second projection line (10) on the second projection line set when no predefined projection lines exist in the first and / or second projection line set. Optical sensing system.

12. In the optical sensing system according to any one of claims 9 to 11, The optical sensing system (100) further includes scanning means, which are adapted to scan the light beam (13) along a trajectory (16) on the scene (2), the trajectory (16) includes a plurality of points (14, 15) on the scene (2), Optical sensing system.

13. In the optical sensing system according to any one of claims 9 to 12, The optical sensing system (100) is adapted to sequentially fill the memory elements with projection lines. Optical sensing system.

14. In the optical sensing system according to any one of claims 9 to 13, The pixels (11) are arranged in M ​​rows and N columns, and each projection line (9, 10) or a predetermined number of projection lines (9, 10) corresponds to at least one pixel. Optical sensing system.

15. In the optical sensing system according to any one of claims 9 to 14, The photodetector is a single-photon detector, preferably a SPAD. Optical sensing system.

Images