Temperature compensation system for depth cameras

Abstract

This invention provides a temperature compensation system for a depth camera, comprising a camera and a projector, and including the following modules: a relationship establishment module for storing a pre-set temperature compensation formula, which expresses the relationship between temperature and the principal point offset of the projector; a principal point calculation module for acquiring the temperature of the projector and determining the target principal point at that temperature based on the temperature compensation formula and the temperature; and a temperature compensation module for acquiring a structured light image captured by the camera at the specified temperature and generating a depth image or point cloud based on the target principal point and the structured light image. This invention can calculate the calibrated target principal point based on the actual operating temperature of the projector, thereby modifying the projector's intrinsic parameters to achieve temperature compensation for the depth camera, reducing errors in depth reconstruction, and expanding the working range of the depth camera.

Description

Technical Field

[0001] This invention relates to depth cameras, and more specifically, to a temperature compensation system for a depth camera. Background Technology

[0002] Commonly used projector technologies include DLP (Digital Light Processing), line laser combined with MEMS mirrors or mechanical mirrors, etc.

[0003] With the maturity of LED light sources and DLP technology, DLP projectors have developed rapidly and become a widely used projection method. In 1987, TI invented the DMD device, enabling the application of DLP digital light processing technology worldwide and further propelling the rise of DLP projectors. A DMD device is a binary pulse-width modulation digital light switch, currently the most complex light switching device in the world. Thousands of tiny square lenses are built on a hinge structure above static random access memory to form a DMD. Each lens can turn on or off the light of one pixel. The hinge structure allows the lens to tilt between two states: +10 degrees for "on" and -10 degrees for "off". Due to the projection principle of DLP projectors, a projection system consisting of a DLP projector and a suitable optical lens can be viewed as a "reverse" pinhole camera model.

[0004] Structured light 3D cameras are highly sensitive to minute changes within their optical components. Temperature variations can cause these minute changes, leading to errors in the depth information generated during triangulation. Current methods for addressing these temperature-induced depth errors often employ active thermal control or active heating control to maintain the 3D camera at its initial calibration temperature. However, this approach not only consumes more energy but also significantly increases the size and cost of the 3D camera. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the purpose of this invention is to provide a temperature compensation system for a depth camera.

[0006] The temperature compensation system for a depth camera provided by this invention includes a camera and a projector, and comprises the following modules:

[0007] The relationship establishment module is used to store a pre-set temperature compensation relationship, which expresses the relationship between temperature and the principal point offset of the projector.

[0008] The principal point calculation module is used to obtain the temperature of the projector and determine the target principal point at that temperature based on the temperature compensation formula and the temperature.

[0009] The temperature compensation module is used to acquire the structured light image captured by the camera at the specified temperature, and to generate a depth image or point cloud by performing temperature compensation of the depth camera based on the target principal point and the structured light image.

[0010] Preferably, the relationship establishment module includes the following units:

[0011] An image acquisition unit is used to acquire multiple temperatures of the projector under multiple ambient temperatures, and control the camera to acquire multiple structured light images at each temperature.

[0012] A phase difference calculation unit is used to determine the phase difference values ​​of the structured light images at multiple temperatures, based on a projector at one of the ambient temperatures.

[0013] The relation establishment unit is used to determine the main point offset of the projector at multiple temperatures based on the phase difference value, and then establish a temperature compensation relation that can express the relationship between the temperature and the main point offset of the projector.

[0014] Preferably, the principal point calculation module includes the following units:

[0015] A temperature acquisition unit is used to acquire the temperature of the projector;

[0016] The main point offset calculation unit is used to determine the main point offset of the projector at a given temperature based on the temperature compensation formula and the temperature.

[0017] The target principal point determination unit is used to obtain the principal point standard value and determine the principal point calibration value based on the principal point offset and the principal point standard value, that is, to determine the target principal point.

[0018] Preferably, the temperature compensation module includes the following units:

[0019] The image acquisition unit is used to acquire the structured light image captured by the camera at the temperature;

[0020] A phase value calculation unit is used to calculate the absolute phase value corresponding to the coordinate value of each pixel on the light receiving sensor in the camera based on the structured light image.

[0021] The calibration information update unit is used to acquire the calibration information generated by the pre-calibration, the calibration information including the intrinsic parameter matrix of the projector, and to update the intrinsic parameter matrix according to the principal point calibration value to generate target calibration information;

[0022] The depth reconstruction unit is used to calculate depth information based on the absolute phase value and the target calibration information, and then perform three-dimensional reconstruction to generate the temperature-compensated depth image or point cloud.

[0023] Preferably, the depth reconstruction unit includes the following components:

[0024] The corresponding pixel calculation unit is used to calculate the corresponding pixel coordinate value on the projector using the absolute phase value;

[0025] The depth information calculation unit is used to determine the distance between the optical center of the light receiving sensor in the depth camera and the coordinates of each object point along the optical axis of the light receiving sensor, based on the coordinate values ​​of two pixels matched by the light receiving sensor and the projector and the target calibration information, thereby generating the depth information of each pixel.

[0026] The depth reconstruction unit is used to perform three-dimensional reconstruction based on the depth information of each pixel to generate a temperature-compensated depth image or point cloud.

[0027] Preferably, the principal point of the projector is the position of the origin of the projector's image coordinate system in the pixel coordinate system;

[0028] The principal point offset of the projector is the offset of the principal point of the projector along the U-axis or V-axis of the pixel coordinate system.

[0029] Preferably, the structured light image is a phase-shift coded image, and the principal point offset is determined based on the average phase difference between the phase-shift coded images acquired at two different temperatures.

[0030] Preferably, the pixel coordinate system is based on the top-left corner of the image as the origin O. P A two-dimensional rectangular coordinate system is established with the row and column directions of the pixels being the U and V coordinate axes, respectively;

[0031] The image coordinate system is a two-dimensional rectangular coordinate system with the intersection of the lens optical axis and the image plane as the origin O1, and the X and Y coordinate axes being parallel to the row and column directions of the pixels.

[0032] Preferably, the temperature compensation formula can be expressed as:

[0033] u 0校准值 =u 0标准值 +(DLP 当前温度 -DLP 标准温度 )×conversion factor, the u 0标准值 DLP 标准温度 The conversion coefficients are pre-stored in the relationship establishment module;

[0034] u 0校准值 The value of u0 in the calibrated principal point coordinates, u 0标准值 The value of u0 in the principal point coordinates at standard temperature, DLP 当前温度 The current ambient temperature during DLP projector operation. 标准温度This refers to the standard ambient temperature for DLP projectors to operate.

[0035] Preferably, the suspension position and orientation of the depth camera remain fixed, and the environment and objects being photographed remain unchanged.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] In this invention, by establishing a relationship between the projector's temperature and the projector's principal point offset, the calibrated target principal point can be calculated based on the actual operating temperature of the projector. This allows for modification of the projector's intrinsic parameters to achieve temperature-based supplementation of the depth camera, reducing errors in depth reconstruction and expanding the working range of the depth camera. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0039] Figure 1 This is a schematic diagram of the module of the temperature compensation method for a depth camera in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the module for constructing the temperature compensation relationship in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the module for determining the target principal point in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of the module for three-dimensional reconstruction in an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of the module for generating depth images or point clouds in an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of the structure of an item picking robot that uses a temperature compensation method based on a depth camera in an embodiment of the present invention.

[0045] Figure 7 This is a flowchart illustrating the steps of the temperature compensation method for a depth camera in an embodiment of the present invention.

[0046] Figure 8 This is a schematic diagram of the temperature compensation device for the depth camera in an embodiment of the present invention; and

[0047] Figure 9 This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Detailed Implementation

[0048] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0049] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0050] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0051] The technical solutions of the present invention and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0052] Figure 1 This is a schematic diagram of the temperature compensation method for a depth camera in an embodiment of the present invention, as shown below. Figure 1 As shown, the temperature compensation method for a depth camera provided by the present invention includes a depth camera and a projector, comprising the following modules:

[0053] The relationship establishment module is used to store a pre-set temperature compensation relationship, which expresses the relationship between temperature and the principal point offset of the projector.

[0054] Figure 2This is a schematic diagram of the module for constructing the temperature compensation relationship in an embodiment of the present invention, such as... Figure 2 As shown, the temperature compensation system for a depth camera provided by the present invention includes the following units in its relationship establishment module:

[0055] An image acquisition unit is used to acquire multiple temperatures of the projector under multiple ambient temperatures, and control the camera to acquire multiple structured light images at each temperature.

[0056] A phase difference calculation unit is used to determine the phase difference values ​​of the structured light images at multiple temperatures, based on a projector at one of the ambient temperatures.

[0057] Tests have shown that the temperature of the projector is significantly correlated with the ambient temperature.

[0058] The relation establishment unit is used to determine the main point offset of the projector at multiple temperatures based on the phase difference value, and then establish a temperature compensation relation that can express the relationship between the temperature and the main point offset of the projector.

[0059] In this embodiment of the invention, the principal point of the projector is the position of the origin of the projector's image coordinate system in the pixel coordinate system;

[0060] The principal point offset of the projector is the offset of the principal point of the projector along the U-axis or V-axis of the pixel coordinate system, preferably along the U-axis of the pixel coordinate system.

[0061] The pixel coordinate system is based on the top left corner of the image as the origin O. P A two-dimensional rectangular coordinate system is established with the row and column directions of the pixels being the U and V coordinate axes, respectively.

[0062] The image coordinate system is a two-dimensional rectangular coordinate system with the intersection of the lens optical axis and the image plane as the origin O1, and the X and Y coordinate axes being parallel to the row and column directions of the pixels.

[0063] The structured light image is a phase-shift coded image, specifically a structured light image with vertical stripes. The principal point offset is determined based on the average phase difference between the phase-shift coded images acquired at two different temperatures.

[0064] The principal point calculation module is used to obtain the temperature of the projector and determine the target principal point at that temperature based on the temperature compensation formula and the temperature.

[0065] Figure 3 This is a schematic diagram of the module for determining the target principal point in an embodiment of the present invention, such as... Figure 3 As shown, the principal point calculation module includes the following units:

[0066] A temperature acquisition unit is used to acquire the temperature of the projector;

[0067] The main point offset calculation unit is used to determine the main point offset of the projector at a given temperature based on the temperature compensation formula and the temperature.

[0068] The target principal point determination unit is used to obtain the principal point standard value and determine the principal point calibration value based on the principal point offset and the principal point standard value, that is, to determine the target principal point.

[0069] In this embodiment of the invention, the projector is a DLP projector, and the temperature of the projector is the system temperature of the DLP projector.

[0070] The principal point offset is determined based on the average phase difference of the coded structured light.

[0071] The temperature compensation module is used to acquire the structured light image captured by the camera at the specified temperature, and to generate a depth image or point cloud by performing temperature compensation of the depth camera based on the target principal point and the structured light image.

[0072] Figure 4 This is a schematic diagram of the module for three-dimensional reconstruction in an embodiment of the present invention, as shown below. Figure 4 As shown, the temperature compensation module includes the following units:

[0073] The image acquisition unit is used to acquire the structured light image captured by the camera at the temperature;

[0074] A phase value calculation unit is used to calculate the absolute phase value corresponding to the coordinate value of each pixel on the light receiving sensor in the camera based on the structured light image.

[0075] The calibration information update unit is used to acquire the calibration information generated by the pre-calibration, the calibration information including the intrinsic parameter matrix of the projector, and to update the intrinsic parameter matrix according to the principal point calibration value to generate target calibration information;

[0076] The depth reconstruction unit is used to calculate depth information based on the absolute phase value and the target calibration information, and then perform three-dimensional reconstruction to generate the temperature-compensated depth image or point cloud.

[0077] In this embodiment of the invention, the intrinsic parameter matrix of the projector is represented as follows:

[0078] Among them, f x ,f y (u0, v0) represents the equivalent focal length of the projector lens, and (u0, v0) represents the principal point coordinates of the projector.

[0079] In this embodiment of the invention, the temperature compensation formula can be expressed as:

[0080] u 0校准值 =u 0标准值 +(DLP 当前温度 -DLP 标准温度 ) × conversion factor. The u 0标准值 DLP 标准温度 Both the conversion coefficient and the conversion factor can be calculated through pre-calibrated temperature settings, or generated through a relation establishment module.

[0081] Among them, u 0校准值 The value of u0 in the calibrated principal point coordinates, u 0标准值 The value of u0 in the principal point coordinates at standard temperature, DLP 当前温度 The current ambient temperature during DLP projector operation. 标准温度 This refers to the standard ambient temperature for DLP projectors to operate.

[0082] Figure 5 This is a schematic diagram of a module for generating depth images or point clouds in an embodiment of the present invention, such as... Figure 5 As shown, the depth reconstruction unit includes the following parts:

[0083] The corresponding pixel calculation unit is used to calculate the corresponding pixel coordinate value on the projector using the absolute phase value;

[0084] The depth information calculation unit is used to determine the distance between the optical center of the light receiving sensor in the depth camera and the coordinates of each object point along the optical axis of the light receiving sensor, based on the coordinate values ​​of two pixels matched by the light receiving sensor and the projector and the target calibration information, thereby generating the depth information of each pixel.

[0085] The depth reconstruction unit is used to perform three-dimensional reconstruction based on the depth information of each pixel to generate a temperature-compensated depth image or point cloud.

[0086] In this embodiment of the invention, the calibration information is calculated and generated during the calibration of the depth camera. Triangulation is used during depth reconstruction.

[0087] In this embodiment of the invention, the position and orientation of the camera suspension are fixed, while the environment and objects being photographed remain unchanged.

[0088] In this embodiment of the invention, the depth map at an ambient temperature of 20°C is used as a reference, and the difference is calculated with the depth maps at ambient temperatures of 0°C, 10°C, 30°C, and 40°C. The average depth difference before and after compensation is obtained. It is determined that the depth image established after temperature compensation has a smaller error. The average depth error before compensation of -2.714 to 2.623 mm can be compensated to -0.827 to 0.797 mm. The average depth error after compensation is reduced to about 30% of that before compensation.

[0089] Figure 6 This is a schematic diagram of the structure of an item picking robot that uses a depth camera temperature compensation method in an embodiment of the present invention, as shown below. Figure 6 As shown, the item picking robot provided by the present invention further includes:

[0090] Unit 1 and Unit 2 are used for storing and / or transporting materials;

[0091] A depth camera 300, whose visual scanning area at least covers the first unit for storing or transporting the material, is used to visually scan the material, acquire a depth image of the material, and generate the pose information and storage location of the material based on the depth image.

[0092] The robot unit 100 is communicatively connected to the depth camera 300, and is used to receive the pose information and storage location, determine the placement state of the target object based on the pose and storage location, and pick up the target object based on the placement state.

[0093] In one embodiment of the present invention, the first unit may be configured as a storage unit 200;

[0094] The storage unit 200 is used to store materials that are placed in an disordered manner. The materials are the target objects, such as metal products, boxes, or any other items.

[0095] The robot unit 100 is communicatively connected to the depth camera 300, and is used to receive the pose information and storage position, determine the placement state of the target object based on the pose and storage position, and pick up the target object according to the placement state and transfer it to the second unit.

[0096] The second unit can be configured to transport or store the picked materials, such as a support rack to facilitate the neat arrangement of items.

[0097] The second unit may also include a transport unit, enabling the robot unit 100 to move the target object on the support frame to the transport unit.

[0098] The depth camera 300 is mounted on the camera bracket 500 and is not shown due to obstruction by the crossbeam of the camera bracket 500.

[0099] The robot unit 100 includes a processor configured to execute the steps of the temperature compensation method for the depth camera by executing executable instructions. By establishing a relationship between the temperature of the projector and the principal point offset of the projector, the processor can calculate the calibrated target principal point based on the actual operating temperature of the projector, thereby modifying the intrinsic parameters of the projector to achieve temperature compensation of the depth camera, reducing errors in depth reconstruction, and expanding the working range of the depth camera.

[0100] Figure 7 This is a schematic diagram of the temperature compensation system for the depth camera in an embodiment of the present invention, as shown below. Figure 7 As shown, the temperature compensation system for a depth camera provided by the present invention, wherein the depth camera includes a camera and a projector, is characterized by comprising the following modules:

[0101] The relationship establishment module is used to store a pre-set temperature compensation relationship, which expresses the relationship between temperature and the principal point offset of the projector.

[0102] The principal point calculation module is used to obtain the temperature of the projector and determine the target principal point at that temperature based on the temperature compensation formula and the temperature.

[0103] The temperature compensation module is used to acquire the structured light image captured by the camera at the specified temperature, and to generate a depth image or point cloud by performing temperature compensation of the depth camera based on the target principal point and the structured light image.

[0104] This invention also provides a temperature compensation device for a depth camera, including a processor and a memory. The memory stores executable instructions for the processor. The processor is configured to execute temperature compensation method steps for the depth camera by executing the executable instructions.

[0105] As shown above, in this embodiment, by establishing a relationship between the projector's temperature and the projector's principal point offset, the calibrated target principal point can be calculated based on the actual operating temperature of the projector. This allows for modification of the projector's intrinsic parameters to achieve temperature supplementation of the depth camera, reducing errors in depth reconstruction and expanding the working range of the depth camera.

[0106] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "platform."

[0107] Figure 8 This is a schematic diagram of the temperature compensation device for a depth camera in an embodiment of the present invention. Refer to the following... Figure 8 To describe an electronic device 600 according to this embodiment of the present invention. Figure 8 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0108] like Figure 8 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0109] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the temperature compensation method section for the depth camera of this specification, according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform, for example... Figure 1 The steps are shown in the figure.

[0110] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0111] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0112] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0113] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, camera, depth camera, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although... Figure 8 As not shown in the diagram, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0114] This invention also provides a computer-readable storage medium for storing a program that, when executed, implements the steps of a temperature compensation method for a depth camera. In some possible implementations, various aspects of the invention can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the above-described temperature compensation method section of this specification according to various exemplary embodiments of the invention.

[0115] As shown above, when the program of the computer-readable storage medium of this embodiment is executed, it establishes a relationship between the temperature of the projector and the principal point offset of the projector, thereby calculating the calibrated target principal point based on the actual operating temperature of the projector, and then modifies the intrinsic parameters of the projector to achieve temperature supplementation of the depth camera, thereby reducing the error in depth reconstruction and expanding the working range of the depth camera.

[0116] Figure 9 This is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. (Reference) Figure 9 As shown, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0117] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0118] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0119] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0120] In this embodiment of the invention, by establishing a relationship between the projector's temperature and the projector's principal point offset, the calibrated target principal point can be calculated based on the actual operating temperature of the projector. This allows for modification of the projector's intrinsic parameters to achieve temperature supplementation of the depth camera, reducing errors in depth reconstruction and expanding the working range of the depth camera.

[0121] The various embodiments described in this specification are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0122] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A temperature compensation system for a depth camera, the depth camera comprising a camera and a projector, characterized in that, The method comprises the following modules: A relationship establishing module is configured to store a preset temperature compensation relationship, which is used to express the relationship between temperature and the principal point offset of the projector; A principal point calculating module is configured to acquire the temperature of the projector, and determine the target principal point at the temperature according to the temperature compensation relationship and the temperature; A temperature compensation module is configured to acquire the structured light image collected by the camera at the temperature, and generate the depth image or point cloud by temperature compensation of the depth camera according to the target principal point and the structured light image; The temperature compensation module comprises the following units: An image acquiring unit is configured to acquire the structured light image collected by the camera at the temperature; A phase value calculating unit is configured to calculate the absolute phase value corresponding to each pixel coordinate value on the light receiving sensor in the camera according to the structured light image; A calibration information updating unit is configured to acquire the calibration information generated by pre-calibration, the calibration information comprises the intrinsic matrix of the projector, and update the intrinsic matrix to generate the target calibration information according to the principal point calibration value; A depth reconstruction unit comprises the following parts: A corresponding pixel calculating part is configured to calculate the corresponding pixel coordinate value on the projector according to the absolute phase value; A depth information calculating part is configured to determine the distance between the optical center of the light receiving sensor in the depth camera and each object point coordinate along the optical axis direction of the light receiving sensor according to the two pixel coordinate values matched between the light receiving sensor and the projector and the target calibration information, that is, to generate the depth information of each pixel; A depth reconstruction part is configured to generate the depth image or point cloud after temperature compensation by three-dimensional reconstruction according to the depth information of each pixel.

2. The temperature compensation system of a depth camera of claim 1, wherein, The relationship establishing module comprises the following units: An image acquiring unit is configured to acquire multiple temperatures of the projector at multiple environmental temperatures, and control the camera to collect multiple structured light images at each temperature; A phase difference calculating unit is configured to determine the phase difference values of the structured light images at multiple temperatures based on the projector at one environmental temperature; A relationship establishing unit is configured to determine the principal point offset of the projector at multiple temperatures according to the phase difference values, and further establish the temperature compensation relationship capable of expressing the relationship between the temperature and the principal point offset of the projector.

3. The temperature compensation system of a depth camera of claim 1, wherein, The principal point calculating module comprises the following units: A temperature acquiring unit is configured to acquire the temperature of the projector; A principal point offset calculating unit is configured to determine the principal point offset of the projector at the temperature according to the temperature compensation relationship and the temperature; A target principal point determining unit is configured to acquire the principal point standard value, and determine the principal point calibration value according to the principal point offset and the principal point standard value, that is, to determine the target principal point.

4. The temperature compensation system of a depth camera of claim 1, wherein, The principal point of the projector is the position of the image coordinate system origin of the projector in the pixel coordinate system; The principal point offset of the projector is the offset of the principal point of the projector along the U-axis or V-axis of the pixel coordinate system.

5. The temperature compensation system of a depth camera of claim 2, wherein, The structured light image is a phase shift encoding image, and the principal point offset is determined according to the average phase difference values of the phase shift encoding images collected at two different temperatures.

6. The temperature compensation system of a depth camera of claim 4, wherein, The pixel coordinate system is a two-dimensional rectangular coordinate system with the upper left corner of the image as the origin O P , and the row and column directions of the pixels are the U and V coordinate axis directions, respectively. The image coordinate system is a two-dimensional rectangular coordinate system with the intersection point of the lens optical axis and the image plane as the origin O1, and parallel to the row and column directions of the pixels as the X and Y coordinate axes.

7. The temperature compensation system of a depth camera of claim 1, wherein, The temperature compensation relationship is expressed as: The u0 calibration value=u0 standard value+(DLP current temperature-DLP standard temperature)×conversion coefficient, the u0 standard value, DLP standard temperature and the conversion coefficient are pre-stored in the relationship establishing module; Wherein, u0 校准值 is the value of u0 in the main point coordinate after calibration, u0 校准值 is the value of u0 in the main point coordinate at standard temperature, DLP 标准温度 is the current ambient temperature when the DLP projector is working, DLP 标准温度 is the standard ambient temperature when the DLP projector is working.

8. The temperature compensation system of a depth camera of claim 1, wherein, The suspension position and posture of the depth camera remain fixed, and the photographed environment and objects remain unchanged.

Images