converting a coordinate system of a three-dimensional camera into an entry point of a two-dimensional camera

Abstract

An example method includes obtaining a three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera; obtaining a two-dimensional coordinate system associated with a two-dimensional camera and a second reference point associated with the two-dimensional camera; aligning the three-dimensional coordinate system with the two-dimensional coordinate system based on a fixed and known positional relationship between the first reference point and the second reference point to obtain a fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; and converting the three-dimensional coordinate system to a reference point of the two-dimensional coordinate system using the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 62 / 957,251, filed January 5, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention generally relates to distance measurement, and more specifically to transforming the coordinate system of a three-dimensional camera into the incident point of a two-dimensional camera. Background Technology

[0004] Two-dimensional images captured by two-dimensional (e.g., red, green, blue, or RGB) cameras are frequently used in applications including object recognition, measurement, autonomous navigation, robotics, and motion capture. In many of these applications, it is useful to link three-dimensional image information to objects or points in these two-dimensional images. Summary of the Invention

[0005] In one example, a method executed by a processing system including at least one processor includes: acquiring a three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera; acquiring a two-dimensional coordinate system associated with a two-dimensional camera and a second reference point associated with the two-dimensional camera; aligning the three-dimensional coordinate system with the two-dimensional coordinate system based on a fixed and known positional relationship between the first reference point and the second reference point to obtain a fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; and using the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, converting the three-dimensional coordinate system into a reference point of the two-dimensional coordinate system.

[0006] In another example, a non-transitory machine-readable storage medium is encoded with instructions executable by a processing system including at least one processor. When executed, the instructions cause the processing system to perform operations including: acquiring a three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera; acquiring a two-dimensional coordinate system associated with a two-dimensional camera and a second reference point associated with the two-dimensional camera; aligning the three-dimensional coordinate system with the two-dimensional coordinate system based on a fixed and known positional relationship between the first and second reference points to obtain a fixed positional relationship between the three-dimensional and two-dimensional coordinate systems; and using the fixed positional relationship between the three-dimensional and two-dimensional coordinate systems, transforming the three-dimensional coordinate system into a reference point of the two-dimensional coordinate system.

[0007] In another example, an apparatus includes a processing system and a non-transitory machine-readable storage medium, the processing system including at least one processor, the non-transitory machine-readable storage medium being encoded with instructions executable by the processing system. When executed, the instructions cause the processing system to perform operations including: acquiring a three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera; acquiring a two-dimensional coordinate system associated with a two-dimensional camera and a second reference point associated with the two-dimensional camera; aligning the three-dimensional coordinate system with the two-dimensional coordinate system based on a fixed and known positional relationship between the first and second reference points to obtain a fixed positional relationship between the three-dimensional and two-dimensional coordinate systems; and using the fixed positional relationship between the three-dimensional and two-dimensional coordinate systems, transforming the three-dimensional coordinate system into a reference point of the two-dimensional coordinate system. Attached Figure Description

[0008] Figure 1 This is a schematic diagram illustrating the relationship between an exemplary distance sensor, including a two-dimensional imaging sensor and a three-dimensional imaging sensor, according to the present disclosure in different coordinate systems;

[0009] Figure 2 The relationship between the exemplary two-dimensional coordinate system and the exemplary three-dimensional reference position is shown in more detail;

[0010] Figure 3 This illustrates the concept of transforming a three-dimensional coordinate system into the front node of a two-dimensional camera;

[0011] Figure 4 The diagram illustrates the coordinate system transformation for a distance sensor whose light-receiving system includes a 3D camera integrated with a 2D camera;

[0012] Figure 5 The diagram illustrates the coordinate system transformation for a distance sensor whose light-receiving system comprises separate (non-integrated) 3D and 2D cameras;

[0013] Figure 6 An exemplary system including multiple 3D cameras is shown;

[0014] Figure 7 This is a flowchart illustrating an exemplary method for transforming the coordinate system of a 3D camera into the incident point of a 2D camera;

[0015] Figure 8 A high-level block diagram depicting an exemplary electronic device for transforming the coordinate system of a 3D camera into the incident point of a 2D camera;

[0016] Figures 9A-9C Various views of a three-dimensional camera are shown according to various aspects of this disclosure; and

[0017] Figure 10It shows a device designed to connect to Figures 9A-9C An isometric view of the structure of a 3D camera from the top of a 2D camera. Detailed Implementation

[0018] This disclosure broadly describes an apparatus, method, and non-transitory computer-readable medium for transforming the coordinate system of a 3D camera to the incident point of a 2D camera. As described above, 2D images captured by a 2D (e.g., red, green, blue, or RGB) camera are frequently used in applications including object recognition, measurement, autonomous navigation, robotics, and motion capture. In many of these applications, it is useful to link 3D image information to objects or points in these 2D images.

[0019] Most 3D sensor systems operate under the premise that the system consists of multiple sensors and cameras, and are therefore configured to use data acquired under these conditions (e.g., 2D images, 3D maps, etc.) to establish the position of the 3D coordinate system. These sensor systems typically do not consider the mechanical positional relationships of the coordinate systems of each sensor or combinations of these coordinate systems. Furthermore, from this perspective, there are no known systems that focus on mechanical positioning. However, the concept of mechanical positioning is crucial when applied to real-world scenarios (e.g., 3D measuring devices that must obtain accurate measurement results).

[0020] This disclosure provides an apparatus for transforming the coordinate system of a 3D camera into the incident point of a 2D camera. In one example, a first reference point is defined as having a fixed, known position relative to the reference point (origin) of the 3D coordinate system associated with the 3D camera. Additionally, a second reference point is defined as having a fixed, known position relative to the reference point (origin) of the 2D coordinate system associated with the 2D camera. The fixed positional relationship between the first and second reference points is also known. Therefore, knowing these three positional relationships allows the transformation of the 3D coordinate system into the foreground node of the 2D camera. By calculating the translation and rotational movements required to achieve the transformation, points in the 3D coordinate system can be assigned position and depth on the imaging sensor of the 2D camera.

[0021] The examples in this disclosure demonstrate that a reference (e.g., a reference point in the two-dimensional coordinate system of a two-dimensional camera) can be matched with a reference (or datum) point in the three-dimensional coordinate system of a three-dimensional camera during calibration. This capability can be used to guide the positioning of the two-dimensional camera. Therefore, the positional relationships of the optical system of the two-dimensional camera (e.g., the optical axis, principal point position, etc., determined by lenses, imaging sensors, and other camera components) relative to the positioning system or mounting mechanism of the three-dimensional camera can be determined under controlled conditions. This allows for the correct linking of two-dimensional and three-dimensional data.

[0022] Other examples of this disclosure provide housing structures for 3D and 2D cameras that allow the 3D and 2D cameras to be connected to form a single device. By using a positioning system on the housing to connect the cameras to the single device, the correspondence between the separate coordinate systems of the 3D and 2D cameras can be easily and correctly achieved. This arrangement also allows the coordinate references of the housing to be pre-set or measured for each 3D and 2D camera, even if the camera specifications are different. In the context of this disclosure, "2D image" is understood to refer to an image acquired using light in the visible spectrum (e.g., by a conventional red, green, and blue (RGB) image sensor). In contrast, images of 3D patterns are acquired using light in the invisible spectrum (e.g., by an infrared imaging sensor).

[0023] This disclosure envisions different configurations of distance sensors that include both two-dimensional and three-dimensional imaging sensors. For example, one type of distance sensor may include two separate light-receiving systems / cameras, wherein a first light-receiving system includes an imaging three-dimensional sensor, and a second light-receiving system includes a two-dimensional camera. In this case, the mechanical positional relationship between the housing of the second light-receiving system and the two-dimensional coordinate system is fixed at a first value. The mechanical positional relationship between positioning and fixing the housing of the second light-receiving system to an external mounting mechanism of the distance sensor and the two-dimensional coordinate system associated with the two-dimensional imaging sensor is fixed at a second value. The first and second values ​​may be stored in a processor-accessible memory of the distance sensor (e.g., the distance sensor's local memory, an external database, etc.).

[0024] The first optical receiving system includes a designated reference point, the position of which relative to a three-dimensional coordinate system is known and fixed at a third value. The reference point is fixed at a fourth value relative to the position of the housing of the first optical receiving system used to fix it to an external mounting mechanism of the distance sensor. Like the first and second values, the third and fourth values ​​can be stored in a processor-accessible memory of the distance sensor. The processor is also capable of transforming the three-dimensional coordinate system to arbitrary positions.

[0025] Another type of distance sensor may include a single integrated light receiving system comprising both a two-dimensional imaging sensor and a three-dimensional imaging sensor. In this case, the distance sensor may include an external mounting mechanism, wherein the position of the external mounting mechanism is determined and managed relative to both the three-dimensional coordinate system associated with the three-dimensional imaging sensor and the two-dimensional coordinate system associated with the two-dimensional imaging sensor.

[0026] Figure 1This is a schematic diagram illustrating the relationship between different coordinate systems of an exemplary distance sensor 100, including a two-dimensional imaging sensor and a three-dimensional imaging sensor, according to the present disclosure. Some components of the distance sensor 100 can be configured in a manner similar to that described in the distance sensors in U.S. Patent Application Serial Nos. 14 / 920,246, 15 / 149,323, and 15 / 149,429.

[0027] For example, as shown, the distance sensor 100 may include a light projection system 102, a light receiving system 104, and a processor 106. The light projection system 104 is configured to project a pattern 108 onto a surface or object 110, wherein the pattern 108 includes a plurality of light spots. The light spots may be arranged in a grid, such as... Figure 1 As shown (e.g., arranged in multiple rows and columns). The rows and columns of the grid can be aligned collinearly or staggered. The light spots may be invisible to the human eye, but visible to the imaging sensor of the distance sensor 100 (as discussed in further detail below).

[0028] Therefore, the points of the three-dimensional pattern 108 can be arranged in a first coordinate system defined by a first axis 112 and a second axis 114 perpendicular to the first axis 112. The first coordinate system may include a three-dimensional “mapped” coordinate system of the distance sensor 100. The first coordinate system may include a reference point 116 defined at the intersection of the first axis 112 and the second axis 114.

[0029] For this purpose, the light projection system 102 may include one or more laser sources capable of projecting a light beam at a wavelength substantially invisible to the human eye (e.g., infrared wavelength). The light projection system 104 may also include one or more diffractive optical elements for splitting the light beam into additional beams. When each beam is incident on the surface or object 110, dots of pattern 108 are generated on the surface or object 110.

[0030] The light receiving system 104 may include an imaging sensor 118 (also more broadly referred to herein as a "camera") for capturing images. The imaging sensor 118 may be a complementary metal-oxide-semiconductor (CMOS) sensor. The images may include a two-dimensional image of the surface or object 110 and an image of a three-dimensional pattern 108 on the surface or object 110. Thus, in one example where the light receiving system 104 includes a single imaging sensor to capture both the two-dimensional image and the image of the three-dimensional pattern 108, the light receiving system 104 may also include a bandpass filter. In this case, when capturing the two-dimensional image (which can be acquired using illumination from the same light source used to generate the pattern 108, such as an infrared source), a bandpass filter may be needed to remove ambient light.

[0031] In an example where the two-dimensional image of surface or object 110 and the image of three-dimensional pattern 108 are acquired by the same imaging sensor, automatic correspondence between positions in the images can be obtained using a first coordinate system associated with the first axis 112, the second axis 114, and the reference point 116. For example, point 120 in three-dimensional pattern 108 can have a position (x, y) in the first coordinate system including the reference point 116. a y a , z a However, in the second two-dimensional coordinate system of the imaging sensor 118, point 120 can have a position (sx). a 1, sy a 1). The fixed positional relationship between the first coordinate system and the second coordinate system (indicated by arrow 122) can be defined as the relationship between the reference point 116 of the first coordinate system and the reference point 124 (e.g., a mechanical reference point) on the distance sensor 100.

[0032] The processor 106 can be configured to control the light projection system 102 to project a three-dimensional pattern 108 and illuminate the surface or object 110 for image capture. The processor 106 can also control the light receiving system 104 to capture a two-dimensional image of the surface or object 110 and an image of the three-dimensional pattern 108. The processor 106 can also perform operations to align the two-dimensional image of the surface or object 110 with the image of the three-dimensional pattern 108.

[0033] Figure 2 The relationship between the exemplary two-dimensional coordinate system and the exemplary three-dimensional reference position is illustrated in more detail. For example... Figure 2 As shown, the reference point 200 of the two-dimensional image coordinate system can be fixed at the front node of the lens 202 of the camera in the light receiving system. As shown, the two-dimensional image position of the mechanical reference point 204 can be measured relative to the reference point 200.

[0034] Figure 2 The diagram also illustrates the effect when the optical axis of the light receiving system is tilted at an angle θ, as indicated by arrow 206. Surface 208 corresponds to the optical axis oriented vertically (e.g., at a 90-degree angle), while surface 208' corresponds to the optical axis rotated by an angle θ.

[0035] The orientation angles in a two-dimensional coordinate system (e.g., the tilt of the z-axis and / or the rotation of the field of view about the z-axis) can be known relative to the two-dimensional coordinate system (e.g., through calibration or some other means).

[0036] Furthermore, the mechanical positional relationship between the two-dimensional coordinate system and the housing of the two-dimensional camera is fixed at a certain value. Therefore, the position of the external mounting mechanism that positions and fixes the housing to the distance sensor relative to the two-dimensional coordinate system is fixed. The orientation angle relative to the two-dimensional coordinate system and the position of the housing relative to the two-dimensional coordinate system can be stored in a processor-accessible memory of the distance sensor.

[0037] Figure 3 This illustrates the concept of transforming a three-dimensional coordinate system into the front node of a two-dimensional camera. Figure 3 Example 300 includes a light projection system 302 for projecting multiple light beams (including beam 304) and a light receiving system, the light receiving system including separate three-dimensional camera 306 (for capturing images of a three-dimensional pattern projected by the light projection system 302, which may be invisible to the human eye) and two-dimensional camera 308 (for capturing two-dimensional images of a surface 310 on which the three-dimensional pattern is projected).

[0038] The coordinate system of the 3D camera 306 includes reference point 312, while the coordinate system of the 2D camera 308 includes reference point 314 (which is also the front node of the lens of the 2D camera 308).

[0039] The coordinate system of the 3D camera 306 can be fixed relative to the imaging sensor of the 3D camera. The coordinate system of the 3D camera 306 can be determined by a calibration process that stores the relationship between the following: (1) the position of the object relative to the distance sensor 300, and (2) the position of the point in the 3D image captured by the 3D camera 306.

[0040] The coordinate system of the 2D camera 308 can be defined as follows: the z-axis can be defined as a line passing through the center of the imaging sensor of the 2D camera and also through the corresponding point in the 2D image captured by the 2D camera 308. The x and y axes can be defined along the pixel array direction of the imaging sensor of the 2D camera.

[0041] The (x, y) and (z) coordinates of point 316 created by beam 304 on surface 310 are shown in a three-dimensional coordinate system, while the corresponding (x, y) coordinates of point 316 created by beam 304 on surface 310 are shown in a two-dimensional coordinate system. c y c ) and (z c )coordinate.

[0042] In one example, the location (p) of point 316 on the imaging sensor of the 2D camera 308. x p y ) can be calculated as:

[0043] (Equation 1)

[0044] Among them, fc It is the focal length of the 2D camera 308 (i.e., the distance between the lens and the imaging sensor), while (x c y c , z c Point 316 is the position of point 316 in the three-dimensional coordinate system associated with the three-dimensional camera 306 (where the origin is transformed into the incident point 314 of the two-dimensional camera 308). Arrow 318 indicates the coordinate transformation between the three-dimensional coordinate system and the two-dimensional coordinate system.

[0045] Or, in another way:

[0046] (Equation 2)

[0047] as well as

[0048] (Equation 3).

[0049] Therefore, each 3D image or "map" captured by the 3D camera 306 can be considered as a point (p) on a 2D image. x p y ) and having depth z c point.

[0050] In one example, the transformation of the coordinate system reference position (e.g., from reference point 312 in the three-dimensional coordinate system to reference point 314 in the two-dimensional coordinate system) can be obtained by the following translation:

[0051] (Equation 4)

[0052] Among them, (T) x T y T z ) is the amount of movement required to translate the reference position of the coordinate system.

[0053] The rotation element of a coordinate system transformation can be defined as:

[0054] (Equation 5)

[0055] Where (x, y, z) is generated by rotating α about x; (x', y', z') is generated by rotating β about y'; and (x', y', Z) is generated by rotating γ about Z.

[0056] exist Figure 3The right side also shows mechanical reference points 322 and 324 for the 2D camera 308 and the 3D camera 306, respectively. Mechanical reference points 322 and 324 are fixed at known positions relative to the 2D and 3D coordinate systems, respectively. For example, the distance d1 (along the y-axis in the 2D coordinate system) between reference point 314 and reference point 320, and the distance d2 (along the x-axis in the 2D coordinate system) between reference point 314 and reference point 320, are fixed. Similarly, the distance d3 (along the y-axis in the 3D coordinate system) between reference point 312 and reference point 322, and the distance d4 (along the x-axis in the 3D coordinate system) between reference point 314 and reference point 320, are fixed. By fixing the 2D camera 304 and the 3D camera 306 in a mechanically determined manner, the positional relationship between the 2D image captured by the 2D camera 304 and the image captured by the 3D camera 306 in the 3D coordinate system can be determined, such as the reference... Figure 7 This will be discussed in more detail. In other words, if the 2D camera 308 has a mechanical reference point 320 at a defined position relative to the reference point 314, then the positional relationship between the 2D coordinate system and the 3D coordinate system can be determined by the mechanical mounting of the 3D camera 306 and the 2D camera 308.

[0057] Since the optical axis of the 2D camera 308 may change due to the precision of the lens assembly, in one example, the 2D camera 308 can be calibrated relative to the mechanical reference point 320.

[0058] It should be noted that the position of the two-dimensional coordinate system relative to the reference point 320 may change due to a change in the incident point of the lens of the two-dimensional camera 304 (e.g., due to focusing or zooming of the lens). In one example, the processor may be able to detect when the incident point changes and adjust the three-dimensional coordinate system to two-dimensional coordinates in a manner that takes into account the change in the incident point. For example, the processor may calculate and store values ​​reflecting changes in the specifications of the two-dimensional camera 304 (e.g., magnification, distortion, etc.) caused by the change in the incident point.

[0059] Furthermore, as described above, the coordinate system of the 3D camera 306 can be determined through a calibration process that stores the relationship between: (1) the position of the object relative to the distance sensor 300, and (2) the position of the point in the 3D image captured by the 3D camera 306. Therefore, in principle, the mechanical reference of the distance sensor 300 corresponds to the coordinate system of the 3D camera 306. However, in order to utilize this correspondence, when the 3D camera 306 is mounted to the distance sensor 300, the mechanical reference should be matched with the positioning device.

[0060] In an exemplary calibration process, the object is relative to reference point O. c(For example, the position of a coordinate system reference point, such as reference point 312) can be set to a known position (e.g., z1, z2, ..., z...). n Then, the position of point a in the three-dimensional coordinate system can be defined as (x... a y a , z a ), for example Figure 1 As shown. The coordinates of point a in the captured image on the imaging sensor of the 2D camera can be defined as (sx... a 1, sy a 1), also as Figure 1 As shown. Therefore, if the object's position is z2, ..., z... n And if the coordinates of the object in the captured image on the imaging sensor of the 2D camera are (sx) a 2, sy a 2), ..., (sx) a n, sy a If n), then the calibration process can be mapped to z1, z2, ..., z n and (sx) a 1, sy a 1), (sx) a 2, sy a 2), ..., (sx) a n, sy a The relationship between n).

[0061] Figure 4 The coordinate system transformation of a distance sensor 400 is illustrated. The light-receiving system of the distance sensor 400 includes a three-dimensional camera 416 integrated with a two-dimensional camera 404. In this case, the distance sensor 400 may include a light projection system 402, a two-dimensional camera 404, and a three-dimensional camera 416. The distance sensor 400 may include additional components, such as a processor, memory, and other components, which are omitted from the illustration for simplicity. The light projection system 402 is configured to project a three-dimensional pattern 406 onto a surface or object 408, wherein the pattern 406 includes a plurality of light points arranged in a grid as described above. The points of the three-dimensional pattern 406 may be arranged in a coordinate system defined by a first axis 412 and a second axis 414 perpendicular to the first axis 412. A reference point 410 may be defined at the intersection of the first axis 412 and the second axis 414.

[0062] The light receiving system includes a two-dimensional camera 404 for capturing two-dimensional images and a three-dimensional camera 416 for capturing images of a three-dimensional pattern 406. The two-dimensional camera 404 and the three-dimensional camera 416 can be integrated (e.g., as a two-dimensional camera including a three-dimensional imaging sensor). In this case, the positional relationship between the three-dimensional coordinate system associated with reference point 410 and the mechanical reference point 418 on the mechanical base of the light receiving system (indicated by arrow 422) is known. The positional relationship between the front node 420 of the two-dimensional camera 404 (which also serves as the reference point for the two-dimensional coordinate system) and the mechanical reference point 418 (indicated by arrow 424) is also known. The transformation from the three-dimensional coordinate system associated with reference point 410 to the two-dimensional coordinate system associated with reference point 420 is indicated by arrow 426.

[0063] Figure 5 The coordinate system transformation of the distance sensor 500 is shown. The light receiving system of the distance sensor 500 includes separate (non-integrated) three-dimensional camera 516 and two-dimensional camera 504. Figure 5 The distance sensor 500 is similar to Figure 4 The distance sensor shown, except in Figure 5 In this configuration, a 3D camera 516, used to capture images of a 3D pattern 506 (projected by a light projection system 502), is attached to a 2D camera 504, used to capture 2D images of an assembled surface or object 508 (i.e., the 3D camera 516 and the 2D camera 504 are not integrated). In this case, the 3D coordinate system of the 3D camera 516 (with a reference point 510 defined at the intersection of coordinate axes 512 and 514) has a fixed, known position relative to a mechanical reference point of the 3D camera 516. Similarly, the 2D coordinate system of the 2D camera 504 (with a reference point 520 defined at the intersection of coordinate axes 522 and 524) also has a fixed, known position relative to a mechanical reference point of the 2D camera 504.

[0064] Therefore, by mounting the 3D camera 516 such that its mechanical reference point has a known position relative to the mechanical reference point of the 2D camera 504, the positional relationship between the 3D and 2D coordinate systems can be determined, and these coordinate systems can be matched through a transformation process. The same principle can be applied to the case where multiple 3D cameras (each with a corresponding mechanical reference point) are mounted relative to the 2D camera 504.

[0065] Therefore, there are many possible configurations that integrate the capabilities of a 2D camera to capture 2D images and a 3D camera to capture 3D images. For example, a 3D camera and a separate 2D camera can be integrated into a single device as described above. In another example, a separate 3D imaging sensor can be mounted on a 2D camera, also as discussed above. In yet another example, a 3D camera can be used to capture both 2D and 3D images.

[0066] 3D cameras that operate by detecting infrared illumination may include bandpass filters to aid in this process. However, when using a 3D camera to capture both 2D and 3D images, the bandpass filter can be omitted. In this case, the 3D camera can capture 2D images under relatively low-light conditions. However, since the 2D and 3D images are captured by different imaging sensors and different optical systems, it becomes necessary to correct for the parallax between the 2D and 3D coordinate systems.

[0067] Figure 6 The diagram shows multiple 3D cameras 6021-602. m An exemplary system 600 (hereinafter individually referred to as "camera 602" or collectively as "camera 602"). Each 3D camera 602 includes a corresponding light projection system 6041-604 for projecting a pattern of light spots. m (Hereinafter referred to individually as "light projection system 604" or collectively as "light projection system 604") and corresponding imaging sensors 6061-606 for capturing images of the projected pattern. m (Hereinafter referred to individually as "Imaging Sensor 606" or collectively as "Imaging Sensor 606"). By using multiple 3D cameras, the range and detection capability of the distance sensing system can be increased.

[0068] Each distance sensor 602 is associated with a corresponding three-dimensional coordinate system 6081-608. m (Hereinafter referred to individually as "coordinate system 608" or collectively as "coordinate system 608") is associated with it. However, the origin 610 defines the principal coordinate system, to which coordinate system 608 will be aligned.

[0069] If the mounting position of the 3D camera 602 relative to the origin 610 (which serves as a reference point for alignment) is known, the positional relationship between coordinate systems 608 can be automatically derived. Therefore, coordinate systems 608 can be matched with each other.

[0070] Figure 7 This is a flowchart illustrating an exemplary method 700 for transforming the coordinate system of a 3D camera to the incident point of a 2D camera. Method 700 can be, for example, a processing system including at least one processor (e.g., a processing system for a distance sensor, e.g., ...). Figure 1 The processor 106) can execute this. Alternatively, method 700 can be executed by, for example, Figure 8 The method 700 is executed by a processing system of a computing device such as the computing device 800 shown and described in more detail below. For illustrative purposes, method 700 is described as being executed by a processing system.

[0071] Method 700 may begin at step 702. In step 704, the processing system of the distance sensor may acquire a three-dimensional coordinate system associated with the three-dimensional camera and a first reference point associated with the three-dimensional camera. The three-dimensional camera may be part of the distance sensor, which includes a light projection system for projecting a three-dimensional pattern onto an object and a processor for calculating the distance to the object based on the appearance of the three-dimensional pattern in an image captured by the three-dimensional camera. Therefore, the three-dimensional coordinate system may include an (x, y, z) coordinate system.

[0072] As described above, the first reference point can have a fixed position relative to a reference point (or origin) of the three-dimensional coordinate system. In one example, the first reference point can include, for example, a mechanical reference point on the housing of a 3D camera, whose position relative to the reference point of the three-dimensional coordinate system is fixed and known through a calibration process. For example, the mechanical reference point can include the mounting point of the 3D camera. The position of the first reference point can be stored in memory accessible to the processing system.

[0073] In step 706, the processing system acquires a two-dimensional coordinate system associated with a two-dimensional camera (e.g., an RGB camera) and a second reference point associated with the two-dimensional camera. The two-dimensional camera may be part of the same distance sensor as the three-dimensional camera. The two-dimensional camera can capture a two-dimensional image of an object on which a three-dimensional pattern is projected (i.e., where the three-dimensional pattern is not visible in the two-dimensional image). Therefore, the two-dimensional coordinate system may include an (x, y) coordinate system. In one example, the three-dimensional camera and the two-dimensional camera may be manufactured as a single integrated device (e.g., as shown in the image). Figure 4 (As shown in the image). In another example, the 3D camera and the 2D camera can be manufactured separately and then mounted to each other after manufacturing (e.g., as shown in the image). Figure 5 (as shown in the image).

[0074] As described above, the second reference point can have a fixed position relative to the reference point (or origin) of the two-dimensional coordinate system. In one example, the second reference point may include the front node (or incident point) of the lens of the two-dimensional camera. In another example, the second reference point may include another mechanical reference point (e.g., a mounting point) on the two-dimensional camera, whose position relative to the origin of the two-dimensional coordinate system is fixed and known. The position of the second reference point can be stored in memory accessible to the processing system.

[0075] In step 708, the processing system can align the three-dimensional coordinate system with the two-dimensional coordinate system based on a fixed and known positional relationship between the first reference point and the second reference point to obtain a fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system. As described above, both the first reference point and the second reference point can have fixed positions relative to corresponding reference points in the three-dimensional coordinate system and the two-dimensional coordinate system. Furthermore, the first reference point and the second reference point can have a certain fixed positional relationship relative to each other. This fixed positional relationship can be stored in a memory accessible to the processing system.

[0076] Therefore, based on the knowledge of the fixed positional relationship between the first and second reference points, and the knowledge of the fixed positional relationship between the first and second reference points and the reference points of the three-dimensional and two-dimensional coordinate systems, the processing system may be able to align the corresponding reference points of the three-dimensional and two-dimensional coordinate systems. Thus, the alignment of the corresponding reference points aligns the three-dimensional and two-dimensional coordinate systems, and enables the processing system to obtain the fixed positional relationship between the three-dimensional and two-dimensional coordinate systems.

[0077] In step 710, the processing system can use the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system obtained in step 708 to transform the three-dimensional coordinate system into a reference point in the two-dimensional coordinate system. In one example, this transformation may involve calculating a point in the three-dimensional coordinate system (e.g., having position (x, y, z)) to a point on the imaging sensor of the two-dimensional camera with depth (e.g., z) (e.g., having position (p)). x p y The translation and rotation components of the required movement. As transformed, the new position of a point in the three-dimensional coordinate system can be called (x...). c y c , z c Equations 4-5 and above Figure 3 An example is given describing the process of converting a three-dimensional coordinate system into a reference point in a two-dimensional coordinate system.

[0078] Method 700 can end at step 712.

[0079] It should be noted that, although not explicitly specified, some boxes, functions, or operations of method 700 described above may include application-specific storage, display, and / or output. In other words, depending on the specific application, any data, records, fields, and / or intermediate results discussed in method 700 may be stored, displayed, and / or output to another device. Furthermore, the description of determining operations or involving decision-making... Figure 7 The boxes, functions, or operations in a given context do not imply that both branches of a given operation are executed. In other words, depending on the outcome of the given operation, one branch of the operation may not be executed.

[0080] Figure 8 A high-level block diagram depicts an exemplary electronic device 800 for transforming the coordinate system of a 3D camera to the incident point of a 2D camera. Thus, the electronic device 800 can be implemented as a processor of an electronic device or system (e.g., a distance sensor). Figure 1 The processor 106).

[0081] like Figure 8 As shown, the electronic device 800 includes a hardware processor element 802 (e.g., a central processing unit (CPU), microprocessor, or multi-core processor), a memory 804 (e.g., random access memory (RAM) and / or read-only memory (ROM)), a module 805 for converting the coordinate system of a 3D camera to the incident point of a 2D camera, and various input / output devices 806, such as storage devices, including but not limited to tape drives, floppy disk drives, hard disk drives or compact disk drives, receivers, transmitters, displays, output ports, input ports, and user input devices, such as keyboards, keypads, mice, microphones, cameras, laser sources, LED light sources, etc.

[0082] Although a single processor element is shown, it should be noted that the electronic device 800 may employ multiple processor elements. Furthermore, although one electronic device 800 is shown in the figure, if for a particular illustrative example, the method(s) described above are implemented in a distributed or parallel manner—that is, a block or the entire method(s) described above is implemented across multiple or parallel electronic devices—then the electronic device 800 in the figure is intended to represent each of those multiple electronic devices.

[0083] It should be noted that this disclosure may be implemented by machine-readable instructions and / or a combination of machine-readable instructions and hardware, such as using application-specific integrated circuits (ASICs), programmable logic arrays (PLAs) including field-programmable gate arrays (FPGAs), or state machines deployed on hardware devices, general-purpose computers, or any other hardware equivalent. For example, computer-readable instructions related to the above-described methods (one or more) may be used to configure a hardware processor to perform blocks, functions, and / or operations of the above-described methods (one or more).

[0084] In one example, the instructions and data (e.g., machine-readable instructions) of this module or process 805 for transforming the coordinate system of a 3D camera to the incident point of a 2D camera can be loaded into memory 804 and executed by hardware processor element 802 to implement the blocks, functions, or operations discussed above in conjunction with method 700. Furthermore, when the hardware processor executes instructions to perform an "operation," this can include the hardware processor directly performing the operation and / or facilitating, directing, or cooperating with another hardware device or component (e.g., a coprocessor) to perform the operation.

[0085] A processor executing machine-readable instructions related to one or more of the methods described above can be considered a programmable processor or a dedicated processor. Thus, the present disclosure's module 805, which transforms the coordinate system of a three-dimensional camera into the incident point of a two-dimensional camera, can be stored on a tangible or physical (broadly non-transitory) computer-readable storage device or medium, such as volatile memory, non-volatile memory, ROM memory, RAM memory, magnetic or optical drives, devices, or disks. More specifically, a computer-readable storage device can include any physical device that provides the ability to store information, such as data and / or instructions accessible by a processor or electronic device (e.g., a computer or controller of a security sensor system).

[0086] In one example, this disclosure provides a novel physical structure for a 3D camera and a 2D camera, which is shaped to be connected in a manner that aligns the respective coordinate systems of the cameras as described above.

[0087] Figures 9A-9C Various views of the three-dimensional camera 900 according to various aspects of this disclosure are shown. In particular, Figure 9A An isometric view of the top of the 3D camera 900 is shown. Figure 9B A side view of the 3D camera 900 is shown, and Figure 9C An isometric view of the bottom of the 3D camera 900 is shown.

[0088] As shown in the figure, a 3D camera 900 typically includes a light projection system and a light receiving system. The light projection system includes optics 902 for projecting multiple light beams, and the light receiving system includes a camera lens 904 for capturing an image of a pattern formed by the multiple light beams. Both the light projection system and the light receiving system are contained within a common housing 906.

[0089] like Figure 9B As shown, the reference point 908 of the 3D camera is defined on the outside of the housing 906, at the base of the light receiving system. The reference point 908 represents the origin of a three-dimensional (e.g., (x, y, z)) coordinate system.

[0090] like Figure 9C As shown, the bottom outer surface (or plane) 914 of the housing 906 also includes a positioning system to facilitate connection to a properly configured two-dimensional camera (see reference). Figure 10 (To be discussed in further detail). In one example, the positioning system includes a pin 910 and a hole 912. The pin 910 extends outward from the bottom outer surface 914 of the housing 906. The hole 912 defines an opening in the bottom outer surface 914 of the housing 906.

[0091] The positioning system of the 3D camera 900 can be used as a reference point for calibrating the 3D camera 900. Specifically, the reference point 908 can be aligned with the hole 912, such as... Figure 9B As shown.

[0092] Figure 10 It shows a device designed to connect to Figures 9A-9C The figure shows an isometric view of the structure of a 3D camera 900 and a 2D camera 1000 from the top. As shown, the 2D camera 1000 includes a light receiving system, which includes a camera lens 1002 for capturing images of objects. The light receiving system is contained within a housing 1004.

[0093] The housing 1004 includes a base portion 1006 and a raised portion 1008 that rises relative to the base portion. The raised portion 1008 contains circuitry and optics for a camera including a lens 1002. The base portion 1006 includes a flat surface 1010 that includes a positioning system to facilitate connection to a suitably configured 3D camera (e.g., the 3D camera 900 of FIG. 9). In one example, the positioning system includes a pin 1012 and a hole 1014. The pin 1012 extends outwardly from the flat surface 1010 of the base portion 1006. The hole 1014 defines an opening in the flat surface 1010 of the base portion 1006.

[0094] The reference point 1016 of the 2D camera 1000 is defined at the base of the pin 1012. Therefore, the positioning system of the 2D camera 1000 can be measured against the optical specifications of the 2D camera.

[0095] As described above, the structure of the 3D camera 900 and the 2D camera 1000 allows them to be connected to form a single device. Specifically, the 3D camera 900 can be arranged in a row with the 2D camera 900, so that the corresponding positioning systems engage with each other, i.e., the pin 910 of the housing 906 of the 3D camera is inserted into the hole 1014 in the housing 1004 of the 2D camera, and the pin 1012 of the housing 1004 of the 2D camera is inserted into the hole 912 in the housing 906 of the 3D camera.

[0096] Therefore, the positioning system of the 3D camera 900 (including pins 910 and holes 912) mechanically positions the reference point 908 of the 3D coordinate system. 3D measurement data will follow the 3D coordinate system defined by this positioning geometry. Similarly, the positioning system of the 2D camera 1000 (including pins 1012 and holes 1014) is fixed to a known (or easily measurable) positional relationship relative to the optical system of the 2D camera (including lens 1002). The arrangement of the connected 3D camera 900 and 2D camera 1000 allows the 3D coordinate system to be positioned arbitrarily.

[0097] In a conventional 2D camera, the captured image may vary relative to the mechanical position of the 2D camera (due to various factors, including variations in the positional relationships of the lens, imaging sensor, and other components, lens aberrations, etc.). However, the structure provided by connecting the 3D camera 900 and the 2D camera 1000 in the manner described above can compensate for camera lens variations using conventional methods for measuring lens variations in the 2D camera 1000.

[0098] It should be understood that the above-disclosed variations and other features and functions, or alternatives thereof, can be combined into many other different systems or applications. Various alternatives, modifications, or changes that are not currently foreseen or anticipated can subsequently be made, and are also intended to be covered by the appended claims.

Claims

1. A method for distance measurement, comprising: A processing system including at least one processor acquires a three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera, wherein the first reference point includes a mechanical reference point on the housing of the three-dimensional camera; The processing system acquires a two-dimensional coordinate system associated with the two-dimensional camera and a second reference point associated with the two-dimensional camera. The processing system aligns the three-dimensional coordinate system with the two-dimensional coordinate system based on a fixed and known positional relationship between the first reference point and the second reference point, thereby obtaining a fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; and The processing system uses the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system to convert the three-dimensional coordinate system into a reference point of the two-dimensional coordinate system.

2. The method according to claim 1, wherein, The processing system, the 3D camera, and the 2D camera are part of a distance sensor.

3. The method according to claim 2, wherein, The distance sensor also includes a memory for storing the first reference point, the second reference point, and the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system.

4. The method according to claim 1, wherein, The second reference point includes the front node of the lens of the two-dimensional camera.

5. The method according to claim 1, wherein, The first reference point includes the origin of the three-dimensional coordinate system, and the second reference point includes the origin of the two-dimensional reference point.

6. The method according to claim 1, wherein, The 3D camera and the 2D camera are integrated into a single system.

7. The method according to claim 1, wherein, The 3D camera and the 2D camera comprise separate, non-integrated systems.

8. The method according to claim 1, wherein, The three-dimensional coordinate system is determined through a calibration process that stores the relationship between the object's position relative to the three-dimensional camera and the point positions in the three-dimensional image captured by the three-dimensional camera.

9. The method according to claim 1, wherein, The z-axis of the two-dimensional coordinate system is defined as a line that passes through the center of the imaging sensor of the two-dimensional camera and through the corresponding point in the two-dimensional image captured by the two-dimensional camera.

10. The method according to claim 9, wherein, The x-axis and y-axis of the two-dimensional coordinate system are defined along the corresponding pixel array direction of the imaging sensor of the two-dimensional camera.

11. The method according to claim 1, wherein, The conversion includes: The processing system calculates and transforms a point (x, y, z) in the three-dimensional coordinate system into a depth position (p) on the imaging sensor of the two-dimensional camera. x p y The translation and rotation components of the required amount of movement.

12. The method according to claim 11, wherein, p x Calculated as Tan -1 (x) c / z c f c p y Calculated as Tan -1 (y) c / z c f c , (x c y c , z c ) is the position of the point (x, y, z) in the three-dimensional coordinate system when it is transformed into the two-dimensional coordinate system, and f c It is the focal length of the two-dimensional camera.

13. The method according to claim 1, wherein, The alignment includes: Position the 3D camera such that the positioning system on the housing of the 3D camera engages with the positioning system on the housing of the 2D camera.

14. A non-transitory machine-readable storage medium encoded with instructions executable by a processing system including at least one processor, wherein, When executed by the processing system, the instructions cause the processing system to perform operations, the operations including: A three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera are obtained, wherein the first reference point includes a mechanical reference point on the housing of the three-dimensional camera; Obtain the two-dimensional coordinate system associated with the two-dimensional camera and the second reference point associated with the two-dimensional camera; Based on the fixed and known positional relationship between the first reference point and the second reference point, the three-dimensional coordinate system is aligned with the two-dimensional coordinate system to obtain a fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; and Using the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the three-dimensional coordinate system is transformed into the reference point of the two-dimensional coordinate system.

15. A distance sensor, comprising: A processing system, including at least one processor; as well as A non-transitory machine-readable storage medium encoded with instructions executable by the processing system, wherein, when executed, the instructions cause the processing system to perform operations including: A three-dimensional coordinate system associated with a three-dimensional camera and a first reference point associated with the three-dimensional camera are obtained, wherein the first reference point includes a mechanical reference point on the housing of the three-dimensional camera; Obtain the two-dimensional coordinate system associated with the two-dimensional camera and the second reference point associated with the two-dimensional camera; Based on the fixed and known positional relationship between the first reference point and the second reference point, the three-dimensional coordinate system is aligned with the two-dimensional coordinate system to obtain a fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system; and Using the fixed positional relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the three-dimensional coordinate system is transformed into the reference point of the two-dimensional coordinate system.

16. The distance sensor according to claim 15, wherein, The alignment includes: Position the 3D camera such that the positioning system on the housing of the 3D camera engages with the positioning system on the housing of the 2D camera.

17. The distance sensor according to claim 16, wherein, The positioning includes: Insert the pin of the positioning system on the housing of the 3D camera into the hole of the positioning system on the housing of the 2D camera; and Insert the pin of the positioning system on the housing of the 2D camera into the hole of the positioning system on the housing of the 3D camera.

18. The distance sensor according to claim 17, wherein, The first reference point is located near the hole of the positioning system on the housing of the 3D camera.

19. The distance sensor according to claim 18, wherein, The second reference point is located near the pin of the positioning system on the housing of the two-dimensional camera.

Images