Method for moving the coordinate system of a three-dimensional camera to the incident position of a two-dimensional camera

Abstract

An exemplary method includes steps of acquiring a three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera; acquiring a two-dimensional coordinate system associated with a two-dimensional camera and a second reference point associated with the two-dimensional camera; aligning the three-dimensional coordinate system with the two-dimensional coordinate system based on a fixed, known positional relationship between the first reference point and the second reference point to acquire a fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; and moving the three-dimensional coordinate system to a data point in the two-dimensional coordinate system using the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 62 / 957,251, filed on January 5, 2020, the entire content of which is incorporated herein by reference.

[0002] The present invention generally relates to distance measurement, and more specifically to a method of moving the coordinate system of a 3 - D camera to the incident position of a 2 - D camera.

Background Art

[0003] Two - dimensional (e.g., red, green, blue i.e., RGB) images captured by a 2 - D camera are often used in applications including object recognition, measurement, autonomous navigation, robotics, and motion capture.

Summary of the Invention

Problems to be Solved by the Invention

[0004] In many of these applications, it is useful to associate 3 - D image information with objects or points within the 2 - D image.

Means for Solving the Problems

[0005] In one example, a method executed by a processing system including at least one processor includes obtaining a 3 - D coordinate system associated with a 3 - D camera and a first reference point associated with the 3 - D camera; obtaining a 2 - D coordinate system associated with a 2 - D camera and a second reference point associated with the 2 - D camera; based on a fixed and known positional relationship between the first reference point and the second reference point, aligning the 3 - D coordinate system with the 2 - D coordinate system to obtain a fixed positional relationship between the 3 - D coordinate system and the 2 - D coordinate system; and using the fixed positional relationship between the 3 - D coordinate system and the 2 - D coordinate system to move the 3 - D coordinate system to the data points of the 2 - D coordinate system.

[0006] In another example, instructions executable by a processing system including at least one processor are encoded on a non-transitory machine-readable storage medium. If the instructions are executed, the processing system is caused to perform operations including: obtaining a three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera; obtaining a two-dimensional coordinate system associated with a two-dimensional camera and a second reference point associated with the two-dimensional camera; aligning the three-dimensional coordinate system with the two-dimensional coordinate system based on a fixed known positional relationship between the first reference point and the second reference point to obtain a fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; and moving the three-dimensional coordinate system to data points of the two-dimensional coordinate system using the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system.

[0007] In another example, an apparatus includes a processing system including at least one processor and a non-transitory machine-readable storage medium encoded with instructions executable by the processing system. If the instructions are executed, the processing system is caused to perform operations including: obtaining a three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera; obtaining a two-dimensional coordinate system associated with a two-dimensional camera and a second reference point associated with the two-dimensional camera; aligning the three-dimensional coordinate system with the two-dimensional coordinate system based on a fixed known positional relationship between the first reference point and the second reference point to obtain a fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; and moving the three-dimensional coordinate system to data points of the two-dimensional coordinate system using the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9A

Figure 9B

Figure 9C

Figure 10

DETAILED DESCRIPTION OF THE INVENTION

[0009] The present disclosure broadly describes an apparatus, a method, and a non-transitory computer-readable medium for moving the coordinate system of a three-dimensional camera to the incident point of a two-dimensional camera. As described above, two-dimensional images captured by a two-dimensional (e.g., red, green, blue, i.e., RGB) camera are widely used in applications including object recognition, measurement, autonomous navigation, robotics, and motion capture. In many of these applications, it is useful to associate three-dimensional image information with objects or points in these two-dimensional images.

[0010] Most 3D sensor systems operate on the premise that the system includes multiple sensors and cameras, and thus is configured to set the position in a 3D coordinate system using the data obtained under these conditions (e.g., 2D images, 3D maps, etc.). These sensor systems typically do not consider the mechanical positional relationships of the coordinate systems of each sensor or combinations of these coordinate systems. Furthermore, there is no known system that focuses on mechanical position adjustment from this perspective. However, in practical applications such as 3D measuring devices where accurate measurement results are essential, the concept of mechanical position adjustment is extremely important.

[0011] Multiple examples of the present disclosure provide means for moving the coordinate system of a 3D camera to the incident point of a 2D camera. In one example, a first reference point having a known fixed position relative to the data point (origin) of the 3D coordinate system associated with the 3D camera is defined. Also, a second reference point having a known fixed position relative to the data point (origin) of the 2D coordinate system associated with the 2D camera is defined. The fixed positional relationship between the first reference point and the second reference point is also known. Therefore, by knowing these three positional relationships, the 3D coordinate system can be moved to the front nodal point of the 2D camera. By calculating the translational and rotational movements required to perform the movement, the position and depth of the imaging sensor of the 2D camera can be assigned to a point in the 3D coordinate system.

[0012] Multiple examples of the present disclosure show that a reference (such as a reference point of the 2D coordinate system of a 2D camera) can be matched with a reference (or data) point of the 3D coordinate system of a 3D camera during calibration. This ability can be used to guide the position adjustment of the 2D camera. Therefore, the positional relationships of the optical system of the 2D camera (e.g., the optical axis, principal point position, etc., determined by the lens, imaging sensor, and other camera elements) can be determined in a controlled state with respect to the position adjustment system or mounting mechanism of the 3D camera. Thereby, 2D data can be correctly associated with 3D data.

[0013] In further examples of the present disclosure, a housing structure for a three-dimensional camera and a two-dimensional camera is provided that allows the three-dimensional camera and the two-dimensional camera to be connected to form a single device. The association between the separate coordinate systems of the three-dimensional camera and the two-dimensional camera can be easily and accurately realized by connecting the plurality of cameras within the single device using a position adjustment system of the housing. This configuration also allows for presetting or measuring a coordinate reference for the housing for each of the three-dimensional camera and the two-dimensional camera, even if the specifications of the cameras are different. Within the scope of the present disclosure, it is to be understood that a "two-dimensional image" refers to an image obtained using light in a spectrum visible to the naked eye (e.g., by a conventional red, green, blue (RGB) image sensor). In contrast, an image of a three-dimensional pattern is obtained using light in a spectrum not visible to the naked eye (e.g., by an infrared imaging sensor).

[0014] The present disclosure contemplates different configurations of a distance sensor that includes both two-dimensional and three-dimensional imaging sensors. For example, one type of distance sensor may include two separate light receiving systems / cameras, where the first light receiving system includes an imaging three-dimensional sensor and the second light receiving system includes a two-dimensional camera. In this case, the mechanical positional relationship of the housing of the second light receiving system with respect to the two-dimensional coordinate system is fixed at a first value. An external mounting mechanism for adjusting the position of the housing of the second light receiving system and fixing it to the distance sensor, and the mechanical positional relationship of the two-dimensional coordinate system associated with the two-dimensional imaging sensor are fixed at a second value. The first value and the second value may be stored in a memory accessible from the processor of the distance sensor (e.g., the local memory of the distance sensor, an external database, etc.).

[0015] The first light receiving system includes a designated reference point whose position relative to the three-dimensional coordinate system is known and fixed at a third value. The position of the reference point relative to the external mounting mechanism used to fix the housing of the first light receiving system to the distance sensor is fixed at a fourth value. Similar to the first value and the second value, the third value and the fourth value may also be stored in a memory accessible from the processor of the distance sensor. The processor may also be movable to any position in the three-dimensional coordinate system.

[0016] Another type of distance sensor may include a single and integrated light receiving system that includes both a two-dimensional imaging sensor and a three-dimensional imaging sensor. In this case, the distance sensor may include an externally mounted mechanism, and the position of the externally mounted mechanism relative to both the three-dimensional coordinate system associated with the three-dimensional imaging sensor and the two-dimensional coordinate system associated with the two-dimensional imaging sensor is determined and managed.

[0017] FIG. 1 is a schematic diagram showing the relationship between different coordinate systems of an exemplary distance sensor 100 including a two-dimensional imaging sensor and a three-dimensional imaging sensor according to the present disclosure. Some elements of the distance sensor 100 may be configured in a similar manner to the distance sensors described in U.S. Patent Application Nos. 14 / 920,246, 15 / 149,323, and 15 / 149,429.

[0018] For example, the distance sensor 100 may include, as shown, a light projection system 102, a light receiving system 104, and a processor 106. The light projection system 104 is configured to project a pattern 108 onto a surface or an object 110, and the pattern 108 includes a plurality of light points. The plurality of light points may be configured in a grid pattern (e.g., configured in a plurality of rows and a plurality of columns) as shown in FIG. 1. The rows and columns of the grid may be collinearly aligned or may be staggered. The plurality of light points may not be visible to the naked eye but may be recognized by the imaging sensor of the distance sensor 100 (described in more detail below).

[0019] Accordingly, a plurality of points of the three-dimensional pattern 108 may be arranged within a first coordinate system defined by a first axis 112 and a second axis 114 orthogonal to the first axis 112. The first coordinate system may include the three-dimensional "map" coordinate system of the distance sensor 100. The first coordinate system may include a data point 116 defined at the intersection of the first axis 112 and the second axis 114.

[0020] For this purpose, the light projection system 102 may include one or more laser light sources capable of projecting a light beam having a wavelength that is substantially invisible to the naked eye (e.g., an infrared wavelength). The light projection system 104 may also include one or more diffractive optical elements that split the light beam into additional light beams. If each light beam is incident on the surface or object 110, points of the pattern 108 are generated on the surface or object 110.

[0021] The light receiving system 104 may include an imaging sensor 118 (also more broadly referred to herein as a "camera") for imaging an image. The imaging sensor 118 may be a complementary metal oxide semiconductor (CMOS) sensor. The image may include an image of the three-dimensional pattern 108 on the surface or object 110, along with a two-dimensional image of the surface or object 110. Thus, in one example, if the light receiving system 104 includes a single imaging sensor to image both the two-dimensional image and the image of the three-dimensional pattern 108, the light receiving system 104 may also include a bandpass filter. In this case, a bandpass filter may be required to remove ambient light when imaging the two-dimensional image (obtained using illumination by the same light source, e.g., the infrared light source used to generate the pattern 108).

[0022] In an example where the two-dimensional image of the surface or object 110 and the image of the three-dimensional pattern 108 are acquired by the same imaging sensor, an automatic correspondence between positions within the image may be obtained using a first coordinate system associated with the first axis 112, the second axis 114, and the data point 116. For example, a point 120 within the three-dimensional pattern 108 may have a position (x a , y a , z a ) within the first coordinate system that includes the data point 116. However, in the second two-dimensional coordinate system of the imaging sensor 118, the point 120 may have a position (sx a 1, sy a 1). The fixed positional relationship between the first coordinate system and the second coordinate system (indicated by the arrow 122) may be defined as the relationship between the data point 116 of the first coordinate system and a reference point 124 (e.g., a mechanical reference point) on the distance sensor 100.

[0023] Processor 106 may be configured to control the light projection system 102 to project the three-dimensional pattern 108 and illuminate the surface or object 110 for image acquisition. Processor 106 may also be configured to control the light receiving system 104 to acquire a two-dimensional image of the surface or object 110 and an image of the three-dimensional pattern 108. Processor 106 may also be configured to perform an operation of aligning the two-dimensional image of the surface or object 110 with the image of the three-dimensional pattern 108.

[0024] FIG. 2 shows in more detail the relationship between an exemplary two-dimensional coordinate system and an exemplary three-dimensional reference position. As shown in FIG. 2, the data point 200 of the two-dimensional image coordinate system may be fixed to the front nodal point of the lens 202 of the camera of the light receiving system. The two-dimensional image position of the mechanical reference point 204 can be measured relative to the data point 200 as shown.

[0025] FIG. 2 also shows the effect when the optical axis of the light receiving system is tilted by an angle θ as indicated by the arrow 206. The surface 208 corresponds to an optical axis directed vertically (e.g., at an angle of 90 degrees), while the surface 208' corresponds to an optical axis rotated by the angle θ.

[0026] The direction angles in the two-dimensional coordinate system (e.g., the tilt of the z-axis and / or the rotation of the field of view about the z-axis) may be known within the two-dimensional coordinate system (e.g., by calibration or other means).

[0027] Furthermore, the mechanical positional relationship between the two-dimensional coordinate system and the housing of the two-dimensional camera is fixed to some value. Accordingly, the position of the external mounting mechanism for positioning the housing relative to the two-dimensional coordinate system and fixing it to the distance sensor is fixed. The direction angle relative to the two-dimensional coordinate system and the position of the housing relative to the two-dimensional coordinate system may be stored in a memory accessible by the processor of the distance sensor.

[0028] Figure 3 shows the concept of moving a three-dimensional coordinate system to the front node of a two-dimensional camera. Example 300 in Figure 3 includes a light projection system 302 that projects a plurality of light beams (including beam 304), and a receiving system that includes a separate three-dimensional camera 306 (which acquires an image of a three-dimensional pattern that is invisible to the naked eye projected by the light projection system 302) and a two-dimensional camera 308 (which acquires a two-dimensional image of the surface 310 onto which the three-dimensional pattern is projected).

[0029] The coordinate system of the three-dimensional camera 306 includes data points 312, while the coordinate system of the two-dimensional camera 308 includes a data point 314 (which is also the front node of the lens of the two-dimensional camera 308).

[0030] The coordinate system of the three-dimensional camera 306 may be fixed relative to the imaging sensor of the three-dimensional camera. The coordinate system of the three-dimensional camera 306 can be determined by a calibration process that preserves the relationship between (1) the position of an object relative to the distance sensor 300 and (2) the position of points within the three-dimensional image captured by the three-dimensional camera 306.

[0031] The coordinate system of the two-dimensional camera 308 can be defined as follows. The z-axis may be defined as a line passing through the center of the imaging sensor of the two-dimensional camera and also passing through the corresponding points of the two-dimensional image captured by the two-dimensional camera 308. The x and y axes may be defined along the pixel array direction of the imaging sensor of the two-dimensional camera.

[0032] The (x, y) and (z) coordinates of point 316 generated on surface 310 by beam 304 are shown in a three-dimensional coordinate system, while the corresponding (x c ,y c ) and (z c ) coordinates of point 316 are shown in a two-dimensional coordinate system.

[0033] The position (p x ,p y ) of point 316 on the imaging sensor of the two-dimensional camera 308 can, in one example, be calculated as follows.

Equation

[0034] In other words, the following equations hold.

Equation

Equation

[0035] Therefore, each three-dimensional image or "map" captured by the three-dimensional camera 306 can be treated as a point having the points (p x , p y ) on the two-dimensional image and the depth z c .

[0036] In one example, the movement of the coordinate system reference position (e.g., from the data point 312 in the three-dimensional coordinate system to the data point 314 in the two-dimensional coordinate system) is obtained corresponding to the following translation.

Equation

[0037] The rotational component of the coordinate system movement can be defined as follows.

Equation

[0038] On the right side of FIG. 3, mechanical reference points 322 and 324 of the two-dimensional camera 308 and the three-dimensional camera 306 are also shown respectively. The mechanical reference points 322 and 324 are fixed at known positions in the two-dimensional coordinate system and the three-dimensional coordinate system. For example, the distance d1 between the data point 314 and the reference point 320 (along the y-axis of the two-dimensional coordinate system) and the distance d2 between the data point 314 and the reference point 320 (along the x-axis of the two-dimensional coordinate system) are fixed. Similarly, the distance d3 between the data point 312 and the reference point 322 (along the y-axis of the three-dimensional coordinate system) and the distance d4 between the data point 314 and the reference point 320 (along the x-axis of the three-dimensional coordinate system) are fixed. By fixing the two-dimensional camera 304 and the three-dimensional camera 306 in a mechanically determined manner, the positional relationship in the three-dimensional coordinate system between the two-dimensional image captured by the two-dimensional camera 304 and the image captured by the three-dimensional camera 306 can be determined in more detail as described with reference to FIG. 7. In other words, when the two-dimensional camera 308 has the mechanical reference point 320 at a position determined with respect to the data point 314, the positional relationship between the three-dimensional coordinate system of the two-dimensional coordinate system can be determined by the mechanical mounting of the three-dimensional camera 306 and the two-dimensional camera 308.

[0039] Since the optical axis of the two-dimensional camera 308 can vary due to the lens assembly accuracy, in one example, calibration of the two-dimensional camera 308 may be performed with respect to the mechanical reference point 320.

[0040] Note that the position of the two - dimensional coordinate system relative to the reference point 320 can change due to changes in the incident point of the lens of the two - dimensional camera 304 (e.g., due to focusing or zooming of the lens). In one example, the processor of the distance sensor can detect when the incident point changes and align the three - dimensional coordinate system with the two - dimensional coordinates while taking into account the change in the incident point. For example, the processor can calculate and store values that reflect changes in the specifications of the two - dimensional camera 304 (e.g., magnification, distortion, etc.) caused by the change in the incident point.

[0041] Furthermore, as described above, the coordinate system of the three - dimensional camera 306 can be determined by a calibration process that stores the relationship between (1) the object position relative to the distance sensor 300 and (2) the position of points in the three - dimensional image captured by the three - dimensional camera 306. Therefore, in principle, there is a correspondence between the mechanical reference point of the distance sensor 300 and the coordinate system of the three - dimensional camera 306. However, to utilize this correspondence, the mechanical reference and the position adjustment means need to match when the three - dimensional camera 306 is mounted on the distance sensor 300.

[0042] In one example of the calibration process, the position of the object relative to the reference point O c (e.g., coordinate system data points such as data point 312) can be set to known positions (e.g., z1, z2,..., z n ). The position of point a in the three - dimensional coordinate system can thus be defined as, for example, (x a , y a , z a ) as shown in FIG. 1. The coordinates of the image of the point on the imaging sensor of the two - dimensional camera can also be defined as (sx a 1, sy a 1) as shown in FIG. 1. If the position of the object is z2,..., z n and the coordinates of the image of the object on the imaging sensor of the two - dimensional camera are (sx a 2, sy a 2),...,(sx a n, sy a n), then the calibration process is based on z1, z2,..., z n and (sx a 1, sy a1), (sx a 2, sy a 2),..., (sx a n, sy a n) can be mapped.

[0043] Figure 4 shows the movement of the coordinate system of a distance sensor 400 that includes a 3D camera 416 integrated with its own light receiving system 2D camera 404. In this case, the distance sensor 400 may include a light projection system 402, a 2D camera 404, and a 3D camera 416. The distance sensor 400 may include additional elements, such as a processor, memory, and other elements, which are omitted from the illustration for simplicity. The light projection system 402 is configured to project a 3D pattern 406 onto a surface or object 408, and the pattern 406 includes a plurality of light points arranged in a grid as described above. The plurality of points of the 3D pattern 406 may be arranged in a coordinate system defined by a first axis 412 and a second axis 414 orthogonal to the first axis 412. A data point 410 may be defined at the intersection of the first axis 412 and the second axis 414.

[0044] The light receiving system includes a 2D camera 404 that captures a 2D image and a 3D camera 416 that captures an image of the 3D pattern 406. The 2D camera 404 and the 3D camera 416 may be integrated (for example, as a 2D camera including a 3D imaging sensor). In this case, the positional relationship (indicated by arrow 422) between the 3D coordinate system associated with the data point 410 and the mechanical reference point 418 of the mechanical base of the light receiving system is known. The positional relationship (indicated by arrow 424) between the front node 420 of the 2D camera 404 (which also serves as a data point in the 2D coordinate system) and the mechanical reference point 418 is also known. The movement from the 3D coordinate system associated with the data point 410 to the 2D coordinate system associated with the data point 420 is indicated by arrow 426.

[0045] FIG. 5 shows the movement of the coordinate system of a distance sensor 500 in which its light receiving system includes a separate (non-integrated) 3D camera 516 and a 2D camera 504. The distance sensor 500 in FIG. 5 is similar to the distance sensor shown in FIG. 4, but in FIG. 5, a 3D camera 516 that images an image of a 3D pattern 506 (projected by a light projection system 502) is attached to a 2D camera 504 that images (i.e., after assembling a 2D image of a surface or an object 508) and the 3D camera 516 and the 2D camera 504 are not integrated. In this case, the 3D coordinate system of the 3D camera 516 (having a data point 510 defined at the intersection of coordinate axes 512 and 514) has a known fixed position relative to the mechanical reference point of the 3D camera 516. The 2D coordinate system of the 2D camera 504 (having a data point 520 defined at the intersection of coordinate axes 522 and 524) also has a known fixed position relative to the mechanical reference point of the 2D camera 504.

[0046] Therefore, by mounting the 3D camera 516 such that the mechanical reference point of the 3D camera 516 has a known position relative to the mechanical reference point of the 2D camera 504, the positional relationship between the 3D coordinate system and the 2D coordinate system can be known, and the coordinate systems can be aligned by movement processing. The same principle applies when multiple 3D cameras (having multiple mechanical reference points) are mounted with respect to the 2D camera 504.

[0047] Therefore, there are many possible configurations for integrating the capabilities of a 2D camera that images 2D images and a 3D camera that images 3D images. For example, one 3D camera and a separate 2D camera can be integrated into a single device as described above. In another example, a separate 3D imaging sensor can also be mounted on one 2D camera as described above. In yet another example, both 2D images and 3D images can be imaged using a 3D camera.

[0048] A 3D camera that functions by detecting infrared irradiation may include a band-pass filter to assist in detecting infrared irradiation. However, when capturing a 2D image together with a 3D image using the 3D camera, the band-pass filter may be omitted. In this case, the 3D camera may capture a 2D image under relatively weak light conditions. However, since the 2D image and the 3D image are captured by different imaging sensors and different optical systems, it is necessary to correct the parallax between the 2D coordinate system and the 3D coordinate system.

[0049] Fig. 6 shows an exemplary system 600 including a plurality of 3D cameras 6021 to 602 m (hereinafter individually referred to as "camera 602" or collectively referred to as "camera group 602"). Each 3D camera 602 includes respective light projection systems 6041 to 604 m (hereinafter individually referred to as "light projection system 604" or collectively referred to as "light projection system group 604") and respective imaging sensors 6061 to 606 m (hereinafter individually referred to as "imaging sensor 606" or collectively referred to as "imaging sensor group 606") for capturing an image of the projected pattern. By using a plurality of 3D cameras, it is possible to expand the range of the distance detection system and improve the detection ability.

[0050] Each distance sensor 602 is associated with respective 3D coordinate systems 6081 to 608 m (hereinafter individually referred to as "coordinate system 608" or collectively referred to as "coordinate system group 608"). However, the origin 610 defines a main coordinate system that is the target for aligning the coordinate system group 608.

[0051] If the mounting position of the 3D camera 602 relative to the origin 610 (which serves as a reference point for alignment purposes) is known, the positional relationship between the coordinate system groups 608 can be automatically derived. Therefore, the coordinate system groups 608 can be made to match each other.

[0052] FIG. 7 is a flowchart showing an exemplary method 700 for moving the coordinate system of a 3D camera to the incident point of a 2D camera. Method 700 may be executed by a processing system including at least one processor, such as a processing system of a distance sensor (e.g., processor 106 of FIG. 1). Alternatively, method 700 may be executed by a processing system of a computing device, such as computing device 800 shown in FIG. 8 and described in detail below. For example, method 700 will be described as being executed by a processing system.

[0053] Method 700 may begin at step 702. At step 704, the processing system of the distance sensor can obtain a 3D coordinate system associated with the 3D camera and a first reference point associated with the 3D camera. The 3D camera may be part of a distance sensor including an optical projection system that projects a 3D pattern onto an object, and a processor that calculates the distance to the object based on the appearance of the 3D pattern in the image captured by the 3D camera. Thus, the 3D coordinate system may include an (x, y, z) coordinate system.

[0054] As described above, the first reference point may have a fixed position relative to a data point (or origin) of the 3D coordinate system. In one example, the first reference point may include, for example, a known mechanical reference point on the housing of the 3D camera, the position of which relative to the data point of the 3D coordinate system is fixed through a calibration process. For example, the mechanical reference point may include the mounting point of the 3D camera. The position of the first reference point may be stored in a memory accessible by the processing system.

[0055] In step 706, the processing system can obtain a two-dimensional coordinate system associated with a two-dimensional camera (e.g., an RGB camera) and a second reference point associated with the two-dimensional camera. The two-dimensional camera may be part of the same distance sensor as the three-dimensional camera. The two-dimensional camera can capture a two-dimensional image of an object onto which a three-dimensional pattern is projected (i.e., a location where the three-dimensional pattern cannot be seen in the two-dimensional image). Accordingly, the two-dimensional coordinate system may include an (x, y) coordinate system. In one example, the three-dimensional camera and the two-dimensional camera may be manufactured as a single integrated device (as shown, for example, in FIG. 4). In another example, the three-dimensional camera and the two-dimensional camera may be manufactured separately and mounted on each other after manufacture (as shown, for example, in FIG. 5).

[0056] As described above, the second reference point may have a position fixed relative to a data point (or origin) of the two-dimensional coordinate system. In one example, the second reference point may include the front nodal point (or entrance point) of the lens of the two-dimensional camera. In another example, the second reference point may include some other mechanical reference point on the two-dimensional camera (e.g., a mounting point) whose position relative to the origin of the two-dimensional coordinate system is fixed and known. The position of the second reference point may be stored in a memory accessible by the processing system.

[0057] In step 708, the processing system can align the three-dimensional coordinate system with the two-dimensional coordinate system based on the fixed known positional relationship between the first reference point and the second reference point to obtain the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system. As described above, both the first reference point and the second reference point may have positions fixed relative to the data points of each of the three-dimensional coordinate system and the two-dimensional coordinate system. Also, the first reference point and the second reference point may have some fixed positional relationship relative to each other. The fixed positional relationship may be stored in a memory accessible by the processing system.

[0058] In this way, the processing system can align the data points in each of the three-dimensional coordinate system and the two-dimensional coordinate system based on the knowledge of the fixed positional relationship between the first reference point and the second reference point, and the knowledge of the fixed positional relationship between the first reference point in the three-dimensional coordinate system and the data point, and between the second reference point in the two-dimensional coordinate system and the data point. The alignment of each data point thus aligns the three-dimensional coordinate system and the two-dimensional coordinate system, and the processing system can obtain the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system.

[0059] In step 710, the processing system can move the three-dimensional coordinate system to the data points of the two-dimensional coordinate system using the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system obtained in step 708. In one example, the movement may include calculating the translational and rotational components of the amount of movement required to move a point (e.g., having a position (x, y, z)) in the three-dimensional coordinate system to a point (e.g., having a position (p x , p y ) on the imaging sensor of a two-dimensional camera having a certain depth (e.g., z). If the new position of the point in the three-dimensional coordinate system has been moved, it may be referred to as (x c , y c , z c ). An example of the process of moving the three-dimensional coordinate system to the data points of the two-dimensional coordinate system is described in Equations 4-5 and Figure 3 above.

[0060] Method 700 may end at step 712.

[0061] Note that although not explicitly specified, some of the blocks, functions, or operations of the above-described method 700 may include steps of storing, displaying, and / or outputting for a specific application. In other words, any data, records, fields, and / or intermediate results described in method 700 may be stored, displayed, and / or output to another device according to a specific application. Further, the blocks, functions, or operations in FIG. 7 that specify a determination operation or are involved in a decision do not mean that both branches of the determination operation are executed. In other words, one of the branches of the determination operation may not be executed depending on the result of the determination operation.

[0062] FIG. 8 shows a high-level block diagram of an exemplary electronic device 800 for moving the coordinate system of a three-dimensional camera to the incident point of a two-dimensional camera. In this way, the electronic device 800 can be implemented as a processor of an electronic device or system such as a distance sensor (e.g., the processor 106 in FIG. 1).

[0063] As shown in FIG. 8, the electronic device 800 includes a hardware processor element 802, such as a central processing unit (CPU), a microprocessor, or, for example, a multi-core processor, a memory 804, such as a random access memory (RAM) and / or a read-only memory (ROM), a module 805 for moving the coordinate system of the three-dimensional camera to the incident point of the two-dimensional camera, and various input / output devices 806, such as a storage device, a tape drive, a floppy (registered trademark) drive, a hard disk drive, or a compact disk drive, a receiver, a transmitter, a display, an output port, an input port, and user input devices such as a keyboard, a keypad, a mouse, a microphone, a camera, a laser light source, an LED light source, etc.

[0064] Although one processor element is shown, it should be noted that the electronic device 800 may use multiple processor elements. Further, although one electronic device 800 is shown in the figure, if the method(s) as described above are implemented distributively or in parallel as a specific illustrative example, that is, if the blocks of the method(s) or the entire method(s) span multiple or parallel electronic devices, the electronic device 800 in the figure shall represent each of these multiple electronic devices.

[0065] Note that the present disclosure can be implemented by a combination of machine-readable instructions and / or machine-readable instructions using a programmable logic array (PLA) including, for example, an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), or by configuring a hardware processor to execute the blocks, functions, and / or operations of the method(s) as disclosed above using a state machine, a general-purpose computer, or any other hardware equivalent mounted on a hardware device, for example, computer-readable instructions related to the method(s) as described above.

[0066] In one example, instructions and data for the current module or process 805 that move the coordinate system of a three-dimensional camera to the incident point of a two-dimensional camera, for example, machine-readable instructions, can be loaded into the memory 804 and executed by the hardware processor element 802 to realize the blocks, functions, or operations as described above in relation to the method 700. Further, when the hardware processor executes instructions to perform an "operation", this may include the hardware processor directly executing the operation and / or facilitating, instructing, or cooperating with another hardware device or element, such as a coprocessor, to execute the operation.

[0067] A processor that executes machine-readable instructions related to the above-described method(s) can be regarded as a programmed processor or a dedicated processor. Thus, the present module 805 that moves the coordinate system of the 3D camera of the present disclosure to the incident point of the 2D camera can be stored in a tangible, i.e., physical (broadly non-transitory) computer-readable storage device or medium, such as volatile memory, non-volatile memory, ROM memory, RAM memory, magnetic or optical drive, device, or diskette, etc. More specifically, the computer-readable storage device may include any physical device that provides the ability to store information such as data and / or instructions accessed from a processor or electronic device such as a computer or controller of a safety sensor system.

[0068] In one example, the present disclosure provides a novel physical structure of a 3D camera and a 2D camera in a shape that is connected to align each coordinate system of the cameras as described above.

[0069] Figures 9A - 9C show a 3D camera 900 according to various aspects of the present disclosure in various display manners. In particular, Figure 9A shows an isometric view of the top of the 3D camera 900, Figure 9B shows a side view of the 3D camera 900, and Figure 9C shows an isometric view of the bottom of the 3D camera 900.

[0070] As shown in the illustration, the 3D camera 900 generally includes a light projection system including an optical component 902 that projects a plurality of light beams, and a light receiving system including a camera lens 904 that images a pattern formed by the plurality of light beams. The light projection system and the light receiving system are housed within a common housing 906.

[0071] As shown in Figure 9B, a data point 908 of the 3D camera is defined outside the housing 906 at the base of the light receiving system. The data point 908 represents the origin of a 3D coordinate system (e.g., (x, y, z)).

[0072] As shown in FIG. 9C, the bottom outer surface (or flat surface) 914 of the housing 906 further includes a position adjustment system that facilitates connection to a properly configured two-dimensional camera (described in more detail with respect to FIG. 10). In one example, the position adjustment system includes pins 910 and bore holes 912. The pins 910 extend outwardly from the bottom outer surface 914 of the housing 906. The bore holes 912 define an opening in the bottom outer surface 914 of the housing 906.

[0073] The position adjustment system of the three-dimensional camera 900 can serve as a reference point when calibrating the three-dimensional camera 900. In particular, the data point 908 may be aligned with the bore hole 912 as shown in FIG. 9B.

[0074] FIG. 10 shows an isometric view of the upper part of a two-dimensional camera 1000 having a structure designed to be connected to the three-dimensional camera 900 of FIGS. 9A - 9C. As shown, the two-dimensional camera 1000 has a light receiving system including a camera lens 1002 for imaging an object of an image. The light receiving system is housed within a housing 1004.

[0075] The housing 1004 includes a base portion 1006 and a raised portion 1008 that is lifted relative to the base portion. The raised portion 1008 includes a circuit and optical components for the camera including the lens 1002. The base portion 1006 includes a flat surface 1010 that includes a position adjustment system that facilitates connection to a properly configured three-dimensional camera (such as the three-dimensional camera 900 of FIG. 9). In one example, the position adjustment system includes pins 1012 and bore holes 1014. The pins 1012 extend outwardly from the flat surface 1010 of the base portion 1006. The bore holes 1014 define an opening in the flat surface 1010 of the base portion 1006.

[0076] The data point 1016 of the two-dimensional camera 1000 is defined at the base of the pin 1012. Thus, the position adjustment system of the two-dimensional camera 1000 can be compared with the optical specifications of the two-dimensional camera.

[0077] As described above, due to the structures of the three-dimensional camera 900 and the two-dimensional camera 1000, the three-dimensional camera 900 and the two-dimensional camera 1000 can be connected to form a single device. Specifically, by aligning the three-dimensional camera 900 with the two-dimensional camera 900 so that their respective position adjustment systems engage with each other, the pin 910 of the housing 906 of the three-dimensional camera can be inserted into the bore hole 1014 of the housing 1004 of the two-dimensional camera, and the pin 1012 of the housing 1004 of the two-dimensional camera can be inserted into the bore hole 912 of the housing 906 of the three-dimensional camera.

[0078] Therefore, the position adjustment system of the three-dimensional camera 900 (including the pin 910 and the bore hole 912) mechanically adjusts the position of the data point 908 in the three-dimensional coordinate system. The three-dimensional measurement data follows the three-dimensional coordinate system defined by the geometry of this position adjustment. Similarly, the position adjustment system of the two-dimensional camera 1000 (including the pin 1012 and the bore hole 1014) is fixed to the known (or easily measurable) positional relationship with the optical system of the two-dimensional camera (including the lens 1002). Due to the arrangement of the connected three-dimensional camera 900 and two-dimensional camera 1000, the three-dimensional coordinate system can be moved to any position.

[0079] In a conventional two-dimensional camera, the captured image can vary with respect to the mechanical position of the two-dimensional camera due to various factors (including fluctuations in the positional relationships of the lens, imaging sensor, and other components, lens abnormalities, etc.). However, the structure obtained by connecting the three-dimensional camera 900 and the two-dimensional camera 1000 in the above-described manner can compensate for fluctuations in the camera lens by adopting a conventional method for measuring fluctuations in the lens of the two-dimensional camera 1000.

[0080] Many other different systems or applications can be obtained by combining the features and functions disclosed above and other alternative ways. Even if various alternative ways, changes, or modifications that are not currently predicted or envisioned are made in the future, these shall be included in the following claims.

Claims

1. Obtaining, by a processing system including at least one processor, a three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera; Obtaining, by the processing system, a two-dimensional coordinate system associated with a two-dimensional camera and a second reference point associated with the two-dimensional camera; Adjusting the position of the three-dimensional camera by the processing system so that a position adjustment system on the housing of the three-dimensional camera engages with a position adjustment system on the housing of the two-dimensional camera based on a fixed known positional relationship between the first reference point and the second reference point, thereby aligning the three-dimensional coordinate system with the two-dimensional coordinate system and obtaining a fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; Moving, by the processing system, the three-dimensional coordinate system to data points of the two-dimensional coordinate system using the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; A method characterized by including the above.

2. The processing system, the three-dimensional camera, and the two-dimensional camera are part of a distance sensor. The method according to claim 1, characterized by the above.

3. The distance sensor further includes a memory for storing the first reference point, the second reference point, and the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system. The method according to claim 2, characterized by the above.

4. The first reference point includes a mechanical reference point on the housing of the three-dimensional camera. The method according to claim 1, characterized by the above.

5. The second reference point includes the front nodal point of the lens of the two-dimensional camera. The method according to claim 1, characterized by the above.

6. The first reference point includes the origin of the three-dimensional coordinate system, and the second reference point includes the origin of the two-dimensional coordinate system. The method according to claim 1, characterized by the above.

7. The three-dimensional camera and the two-dimensional camera are integrated as a single system. The method according to claim 1, characterized by the above.

8. The three-dimensional camera and the two-dimensional camera include separate non-integrated systems. The method according to claim 1, characterized by the above.

9. The three-dimensional coordinate system is determined by a calibration process that stores the relationship between the object position with respect to the three-dimensional camera and the position of points in the three-dimensional image captured by the three-dimensional camera. The method according to claim 1, characterized in that...

10. wherein the step of moving comprises The processing system calculates a translational component and a rotational component of a movement amount necessary to move a point (x, y, z) in the three-dimensional coordinate system to a position (p ｘ , p ｙ ) of an imaging sensor of a two-dimensional camera having a certain depth. The method according to claim 1, characterized in that...

11. p ｘ is calculated as Tan －１ (x ｃ / z ｃ )f ｃ and p ｙ is calculated as Tan －１ (y ｃ / z ｃ )f ｃ where (px, py) is the position when the point (xc, yc, zc) in the three-dimensional coordinate system is moved to the two-dimensional coordinate system, and f ｃ is the focal length of the two-dimensional camera The method according to claim 10, characterized in that...

12. A non-transitory machine-readable storage medium encoded with instructions executable by a processing system including at least one processor, wherein if the instructions are executed by the processing system, the processing system is caused to obtain a three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera; obtain a two-dimensional coordinate system associated with a two-dimensional camera and a second reference point associated with the two-dimensional camera; based on a fixed known positional relationship between the first reference point and the second reference point, perform position adjustment of the three-dimensional camera so that a position adjustment system on the housing of the three-dimensional camera engages with a position adjustment system on the housing of the two-dimensional camera, thereby aligning the three-dimensional coordinate system with the two-dimensional coordinate system and obtaining a fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; using the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, move the three-dimensional coordinate system to data points of the two-dimensional coordinate system; perform operations including a medium, characterized in that...

13. a processing system including at least one processor; and a non-transitory machine-readable storage medium encoded with instructions executable by the processing system, wherein if the instructions are executed, the processing system is caused to obtain a three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera; obtain a two-dimensional coordinate system associated with a two-dimensional camera and a second reference point associated with the two-dimensional camera; based on a fixed known positional relationship between the first reference point and the second reference point, perform position adjustment of the three-dimensional camera so that a position adjustment system on the housing of the three-dimensional camera engages with a position adjustment system on the housing of the two-dimensional camera, thereby aligning the three-dimensional coordinate system with the two-dimensional coordinate system and obtaining a fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; A step of moving the three-dimensional coordinate system to the data points of the two-dimensional coordinate system by using the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; A medium including an operation including the above to be executed; A distance sensor characterized by the above.

14. The step of performing the position adjustment includes: A step of inserting a pin of a position adjustment system on the housing of the three-dimensional camera into a bore hole of a position adjustment system on the housing of the two-dimensional camera; A step of inserting a pin of a position adjustment system on the housing of the two-dimensional camera into a bore hole of a position adjustment system on the housing of the three-dimensional camera; The distance sensor according to claim 13, characterized by including the above.

15. The first reference point is located in the bore hole of the position adjustment system on the housing of the three-dimensional camera; The distance sensor according to claim 14, characterized by the above.

16. The second reference point is located at the base of the pin of the position adjustment system on the housing of the two-dimensional camera; The distance sensor according to claim 15, characterized by the above.

Images