Depth camera calibration method, device, equipment and storage medium

Abstract

The embodiments of the present disclosure disclose a method, apparatus, device and storage medium for calibrating a depth camera. The method includes: controlling the depth camera to be calibrated to shoot a 3D calibration plate from multiple angles and / or multiple distances to obtain an RGB image and a depth map; extracting the first feature point information in the RGB image and the second feature point information in the depth map; wherein the first feature point information includes the first feature point coordinates and the first depth; the second feature point information includes the second feature point coordinates and the second depth; training a set neural network model based on the first feature point information and the second feature point information to obtain the calibration model of the depth camera. The calibration method for a depth camera provided by the embodiments of the present disclosure trains a set neural network model based on the first feature point information in the RGB image and the second feature point information in the depth map to obtain the calibration model of the depth camera, which can improve the accuracy and efficiency of the depth camera calibration.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and more particularly to a depth camera calibration method, apparatus, device, and storage medium. Background Art

[0002] Traditional depth camera calibration relies on flat markers. For example, a printed chessboard is placed on a flat surface to serve as the calibration marker. By adjusting the orientation of the calibration marker or the camera, photos of the calibration marker are taken from different angles. Feature points are extracted from the photos, and the intrinsic and extrinsic parameters are estimated using the least squares or maximum likelihood methods for an ideal, undistorted image. However, existing calibration methods suffer from poor accuracy and low efficiency. Summary of the Invention

[0003] The embodiments of the present disclosure provide a depth camera calibration method, apparatus, device, and storage medium, which can improve the accuracy and efficiency of depth camera calibration.

[0004] In a first aspect, an embodiment of the present disclosure provides a depth camera calibration method, comprising:

[0005] Control the depth camera to be calibrated to shoot the 3D calibration plate from multiple angles and / or multiple distances to obtain RGB images and depth maps;

[0006] Extracting first feature point information from the RGB image and second feature point information from the depth image; wherein the first feature point information includes first feature point coordinates and a first depth; and the second feature point information includes second feature point coordinates and a second depth;

[0007] A neural network model is trained based on the first feature point information and the second feature point information to obtain a calibration model of the depth camera.

[0008] In a second aspect, an embodiment of the present disclosure further provides a depth camera calibration device, characterized by comprising:

[0009] The RGB image and depth map acquisition module is used to control the depth camera to be calibrated to shoot the 3D calibration plate from multiple angles and / or multiple distances to obtain the RGB image and depth map;

[0010] A feature point information extraction module is configured to extract first feature point information from the RGB image and second feature point information from the depth image; wherein the first feature point information includes first feature point coordinates and a first depth; and the second feature point information includes second feature point coordinates and a second depth;

[0011] A calibration model acquisition module is used to train a set neural network model based on the first feature point information and the second feature point information to obtain a calibration model of the depth camera.

[0012] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0013] one or more processing devices;

[0014] a storage device for storing one or more programs;

[0015] When the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the depth camera calibration method as described in the embodiment of the present disclosure.

[0016] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the depth camera calibration method as described in the embodiment of the present disclosure.

[0017] The embodiments of the present disclosure disclose a method, apparatus, device and storage medium for calibrating a depth camera. The depth camera to be calibrated is controlled to shoot a 3D calibration plate from multiple angles and / or multiple distances to obtain an RGB image and a depth map; the first feature point information in the RGB image and the second feature point information in the depth map are extracted; wherein the first feature point information includes the first feature point coordinates and the first depth; the second feature point information includes the second feature point coordinates and the second depth; based on the first feature point information and the second feature point information, a set neural network model is trained to obtain a calibration model of the depth camera. The calibration method for a depth camera provided by the embodiments of the present disclosure trains a set neural network model based on the first feature point information in the RGB image and the second feature point information in the depth map to obtain a calibration model of the depth camera, which can improve the accuracy and efficiency of the depth camera calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flowchart of a depth camera calibration method provided by an embodiment of the present disclosure;

[0019] Figure 2a is a schematic diagram of a depth camera provided by an embodiment of the present disclosure photographing a 3D calibration plate;

[0020] Figure 2b is a front view of a calibration block provided by an embodiment of the present disclosure;

[0021] Figure 2c is a top view of a calibration block provided by an embodiment of the present disclosure;

[0022] Figure 2dis a side view of a calibration block provided by an embodiment of the present disclosure;

[0023] Figure 2e This is an example image of a 3D calibration plate captured by a depth camera at multiple distances.

[0024] Figure 2f This is an example image of a 3D calibration plate captured by a depth camera from multiple positions;

[0025] Figure 2g It is a schematic diagram of the correspondence between the RGB image and the depth image before calibration;

[0026] Figure 2h It is a schematic diagram of the correspondence between the RGB image and the depth image after calibration;

[0027] Figure 3 is a structural diagram of a depth camera calibration device provided by an embodiment of the present disclosure;

[0028] Figure 4 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0030] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0031] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0032] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0033] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0034] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0035] Depth camera technology has become increasingly mature in recent years. This technology can obtain accurate depth information in the shooting scene at a low cost. This depth information can be subsequently used in multiple businesses, such as three-dimensional data acquisition of virtual reality (VR) real estate, and for next-generation 3D videos. The traditional depth camera marking method is camera calibration based on planar markers. For example, print a chessboard and stick it on a plane as a calibration object. By adjusting the direction of the calibration object or the camera, take some photos of the calibration object in different directions; then extract feature points from the photos; use the least squares method or the maximum likelihood method to estimate the internal and external parameters under ideal distortion-free conditions. However, the existing calibration methods have the problems of poor accuracy and low efficiency. To address this problem, the embodiments of the present disclosure propose a calibration method for a depth camera.

[0036] Figure 1 This is a flow chart of a depth camera calibration method provided by an embodiment of the present disclosure. This embodiment is applicable to the case where a depth camera is used for calibration. The method can be executed by a depth camera calibration device, which can be composed of hardware and / or software and can generally be integrated into a device with a depth camera calibration function, which can be an electronic device such as a server, a mobile terminal or a server cluster. Figure 1 As shown, the method specifically includes the following steps:

[0037] Step 110: Control the depth camera to be calibrated to shoot the 3D calibration plate from multiple angles and / or multiple distances to obtain an RGB image and a depth image.

[0038] Calibration involves using a known calibration object to establish correspondences between points on the object with known coordinates and their image points, and then using a specific algorithm to obtain the intrinsic and extrinsic parameters of the camera model. Before using a depth camera, its intrinsic and extrinsic parameters must be calculated. Using a set of intrinsic and extrinsic camera parameters, image coordinates are mapped to real-world coordinates. A depth camera to be calibrated can be understood as one whose intrinsic and extrinsic parameters are to be obtained.

[0039] Figure 2a FIG is a schematic diagram of a depth camera provided in an embodiment of the present disclosure photographing a 3D calibration plate. Figure 2aAs shown, a 3D calibration plate is equipped with multiple calibration blocks of different heights, each with a black and white pattern printed on it. Unlike a 2D calibration plate, a 3D calibration plate's calibration blocks are cubes of varying sizes and heights, each with a known height, allowing simulation of varying depths. A 3D calibration plate must meet the following requirements: a flat surface and non-light-absorbing material; and an asymmetrical, irregularly distributed pattern on the surface. The black and white pattern can be either flat or three-dimensional.

[0040] Preferably, the black and white pattern is a three-dimensional pattern. To better extract feature points, two-dimensional codes, such as "AprilCode," can be printed on different calibration blocks on the 3D calibration board to facilitate feature point matching. Furthermore, the present disclosure supports three-dimensional printing of "AprilCode," where the cross-section of the calibration block is a pattern corresponding to the AprilCode, and the three-dimensional graphic has a certain height / depth.

[0041] Specifically, the depth camera to be calibrated is controlled to shoot the 3D calibration plate from multiple angles and / or multiple distances. For example, it can be shot from multiple angles such as front view, top view and side view to obtain the front view, top view and side view. For example, Figure 2b is a front view of a calibration block provided by an embodiment of the present disclosure;

[0042] Figure 2c is a top view of a calibration block provided by an embodiment of the present disclosure; Figure 2d This is a side view of a calibration block provided by an embodiment of the present disclosure. In the depth direction, within the range of 80cm-150cm from the plane of the 3D calibration plate, data is collected every 10cm, and the distance between the calibration plate plane and the camera is recorded as D distance In the horizontal and vertical directions, the calibration plate is moved to capture data at different positions. For example, Figure 2e This is an example image of a 3D calibration plate captured by a depth camera at multiple distances. Figure 2f This is an example image of a 3D calibration plate captured by a depth camera from multiple positions.

[0043] Step 120: extract the first feature point information in the RGB image and the second feature point information in the depth image.

[0044] The first feature point information includes the first feature point coordinates and the first depth; the second feature point information includes the second feature point coordinates and the second depth.

[0045] In this embodiment, since the depth map and the RGB image are captured by the same camera, the depth map can be corrected using the RGB image.

[0046] Specifically, the first feature point information in the RGB image and the second feature point information in the depth image may be extracted by extracting the first feature point information of each calibration block in the RGB image and the second feature point information of each calibration block in the depth image. The first feature point and the second feature point are corner points of a black and white pattern. In this embodiment, a black and white pattern is provided on the calibration block, and the corner points of the black and white pattern can be used as feature points of the calibration block.

[0047] Exemplarily, the coordinates of the first feature point may be expressed as PRGB, the coordinates of the second feature point may be expressed as PIR, and the second depth may be extracted from the depth map and expressed as d.

[0048] In this embodiment, the first depth of each calibration block in the RGB image may be extracted by: determining the distance between the optical center of the camera to be calibrated and the 3D calibration plate; and determining the first depth according to the distance and the height of the calibration block.

[0049] The height of each corner point on the calibration block relative to the calibration plate plane can be expressed as D (if the black and white pattern is two-dimensional, it is equivalent to the height of the calibration block; if the black and white pattern is three-dimensional, it is equivalent to the sum of the height of the calibration block and the height of the black and white pattern). Specifically, the distance between the optical center of the camera to be calibrated and the 3D calibration plate is measured and can be expressed as D distance , the height of each corner point on the calibration block relative to the calibration plate plane is known and can be expressed as D. Then the first depth is the difference between the distance between the optical center of the camera to be calibrated and the 3D calibration plate and the actual height of the calibration block, which can be expressed as (D distance -D).

[0050] Step 130: Train a set neural network model based on the first feature point information and the second feature point information to obtain a calibration model of the depth camera.

[0051] The neural network model may be a multi-layer perceptron model, which can map the input data set to the output data set.

[0052] In this embodiment, the calibration model of the depth camera refers to a model for correcting the coordinate information and depth information of each pixel point in the depth map.

[0053] Specifically, the second feature point information is input into the set neural network model to obtain predicted feature point information, the predicted feature point information is compared with the first feature point information to obtain the difference, and the parameters of the set neural network model are adjusted according to the difference to obtain the calibration model of the depth camera.

[0054] Furthermore, a method of training a set neural network model based on the first feature point information and the second feature point information to obtain a calibration model of the depth camera may be:

[0055] a1) Inputting the second feature point information into the set neural network model to obtain predicted feature point information.

[0056] Specifically, the second feature point information includes the second feature point coordinates and the second depth extracted from the RGB image. The second feature point coordinates and the second depth information are combined into a set of data and input into the set neural network model to obtain the coordinates and depth information of the predicted feature point.

[0057] Exemplarily, the second feature point information is expressed as (PIR, d), where PIR is the coordinate of the second feature point, and d is the second depth. (PIR, d) is input into the set neural network model to obtain the output predicted feature point information, which can be expressed as (PIR*, d*), where PIR* is the coordinate information of the predicted feature point, and d* is the depth information of the predicted feature point.

[0058] b1) Determine a loss function between the predicted feature point information and the first feature point information.

[0059] The loss function may also be referred to as a cost function, which may be specifically understood as a function that characterizes the difference between the predicted feature point information and the first feature point information.

[0060] For example, the above example is described, the predicted feature point information can be expressed as (PIR*, d*), where PIR* is the coordinate information of the predicted feature point, and d* is the depth information of the predicted feature point; the first feature point information can be expressed as (PRGB, (D distance -D)), where PRGB is the coordinate information of the first feature point, (D distance -D) is the first depth. The predicted feature point information (PRGB, (D distance -D)) and the error between the predicted feature point information (PIR*,d*).

[0061] c1) Train the set neural network model based on the loss function to obtain the calibration model of the depth camera.

[0062] Specifically, the parameters of the neural network model are adjusted and set according to the loss function until the loss function meets the set conditions, and the depth camera calibration model training is completed.

[0063] Furthermore, after obtaining the calibration model of the depth camera, the method further includes:

[0064] a2) Control the calibrated depth camera to shoot the target scene and obtain an RGB image and an initial depth map.

[0065] Among them, the first feature point information of each calibration block can be extracted from the RGB image, including the first feature point coordinates and the first depth; the second feature point information of each calibration block can be extracted from the initial depth image, including the second feature point coordinates and the second depth.

[0066] b2) Input the initial depth map into the calibration model to obtain the corrected depth map.

[0067] The initial depth map includes feature point coordinate information and depth information; the corrected depth map includes corrected feature point coordinate information and depth information.

[0068] c2) Align the RGB image and the rectified depth image to obtain the RGBD image corresponding to the target scene.

[0069] Specifically, by aligning the pixels between the RGB image and the rectified depth image, the RGBD image corresponding to the target scene can be obtained.

[0070] For example, Figure 2g This is a schematic diagram of the correspondence between the RGB image and the depth image before calibration. The pixel offset can be clearly seen; Figure 2h This is a schematic diagram of the correspondence between the RGB image and the depth map after calibration. It can be seen that the pixels between the RGB image and the depth map are aligned.

[0071] The embodiments of the present disclosure disclose a method, apparatus, device and storage medium for calibrating a depth camera. The depth camera to be calibrated is controlled to shoot a 3D calibration plate from multiple angles and / or multiple distances to obtain an RGB image and a depth map; the first feature point information in the RGB image and the second feature point information in the depth map are extracted; wherein the first feature point information includes the first feature point coordinates and the first depth; the second feature point information includes the second feature point coordinates and the second depth; based on the first feature point information and the second feature point information, a set neural network model is trained to obtain a calibration model of the depth camera. The calibration method for a depth camera provided by the embodiments of the present disclosure trains a set neural network model based on the first feature point information in the RGB image and the second feature point information in the depth map to obtain a calibration model of the depth camera, which can improve the accuracy and efficiency of the depth camera calibration.

[0072] Figure 3 FIG is a schematic diagram of a structure of a depth camera calibration device provided by an embodiment of the present disclosure. Figure 3 As shown, the device includes:

[0073] The RGB image and depth map acquisition module 210 is used to control the depth camera to be calibrated to shoot the 3D calibration plate from multiple angles and / or multiple distances to obtain an RGB image and a depth map;

[0074] The feature point information extraction module 220 is configured to extract first feature point information from the RGB image and second feature point information from the depth image; wherein the first feature point information includes first feature point coordinates and a first depth; and the second feature point information includes second feature point coordinates and a second depth;

[0075] The calibration model acquisition module 230 is used to train a set neural network model based on the first feature point information and the second feature point information to obtain a calibration model of the depth camera.

[0076] Optionally, a plurality of calibration blocks of different heights are provided on the 3D calibration plate, and each calibration block is printed with a black and white pattern.

[0077] Optionally, the black and white pattern is a three-dimensional pattern.

[0078] Optionally, the feature point information extraction module 220 includes:

[0079] The feature point information extraction unit of each calibration block is used to extract the first feature point information of each calibration block in the RGB image and the second feature point information of each calibration block in the depth image; wherein the first feature point and the second feature point are corner points of the black and white pattern.

[0080] Optionally, each calibration block feature point information extraction unit is specifically used to:

[0081] Determine the distance between the optical center of the camera to be calibrated and the 3D calibration plate;

[0082] The first depth is determined based on the distance and the height of the calibration block.

[0083] Optionally, the calibration model acquisition module 230 is specifically configured to:

[0084] Inputting the second feature point information into the set neural network to obtain predicted feature point information;

[0085] Determining a loss function between the predicted feature point information and the first feature point information;

[0086] The set neural network model is trained based on the loss function to obtain the calibration model of the depth camera.

[0087] Optionally, the device further includes:

[0088] The RGB image and initial depth map acquisition module is used to control the calibrated depth camera to shoot the target scene and obtain the RGB image and initial depth map;

[0089] The corrected depth map acquisition module is used to input the initial depth map into the calibration model to obtain the corrected depth map;

[0090] The RGBD image acquisition module is used to align the RGB image and the rectified depth image to obtain the RGBD image corresponding to the target scene.

[0091] The above device can execute the methods provided by all the above embodiments of the present disclosure, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not fully described in this embodiment, please refer to the methods provided by all the above embodiments of the present disclosure.

[0092] Reference below Figure 4 , which shows a schematic structural diagram of an electronic device 300 suitable for implementing the embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), etc., fixed terminals such as digital TVs, desktop computers, etc., or various forms of servers, such as independent servers or server clusters. Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0093] like Figure 4 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory device (ROM) 302 or a program loaded from a storage device 305 into a random access memory device (RAM) 303. Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0094] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 4 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0095] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing a word recommendation method. In such an embodiment, the computer program can be downloaded and installed from a network via the communication device 309, or installed from the storage device 305, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0096] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0097] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0098] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0099] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: controls the depth camera to be calibrated to shoot the 3D calibration plate from multiple angles and / or multiple distances to obtain an RGB image and a depth map; extracts the first feature point information in the RGB image and the second feature point information in the depth map; wherein the first feature point information includes the first feature point coordinates and the first depth; the second feature point information includes the second feature point coordinates and the second depth; and trains the set neural network model based on the first feature point information and the second feature point information to obtain the calibration model of the depth camera.

[0100] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0103] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0104] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0105] According to one or more embodiments of the present disclosure, a depth camera calibration method is disclosed, including:

[0106] Control the depth camera to be calibrated to shoot the 3D calibration plate from multiple angles and / or multiple distances to obtain RGB images and depth maps;

[0107] Extracting first feature point information from the RGB image and second feature point information from the depth image; wherein the first feature point information includes first feature point coordinates and a first depth; and the second feature point information includes second feature point coordinates and a second depth;

[0108] A neural network model is trained based on the first feature point information and the second feature point information to obtain a calibration model of the depth camera.

[0109] Furthermore, the 3D calibration plate is provided with a plurality of calibration blocks of different heights, and each calibration block is printed with a black and white pattern.

[0110] Furthermore, the black and white pattern is a three-dimensional pattern.

[0111] Furthermore, extracting the first feature point information in the RGB image and the second feature point information in the depth image includes:

[0112] Extract first feature point information of each calibration block in the RGB image and second feature point information of each calibration block in the depth image; wherein the first feature point and the second feature point are corner points of the black and white pattern.

[0113] Furthermore, extracting the first depth of each calibration block in the RGB image includes:

[0114] Determine the distance between the optical center of the camera to be calibrated and the 3D calibration plate;

[0115] A first depth is determined according to the distance and the height of the calibration block.

[0116] Furthermore, training a set neural network model based on the first feature point information and the second feature point information to obtain a calibration model of the depth camera includes:

[0117] Inputting the second feature point information into the set neural network to obtain predicted feature point information;

[0118] determining a loss function between the predicted feature point information and the first feature point information;

[0119] The set neural network model is trained based on the loss function to obtain a calibration model of the depth camera.

[0120] Furthermore, after obtaining the calibration model of the depth camera, the method further includes:

[0121] Control the calibrated depth camera to shoot the target scene and obtain the RGB image and initial depth map;

[0122] Inputting the initial depth map into the calibration model to obtain a corrected depth map;

[0123] The RGB image and the corrected depth image are aligned to obtain an RGBD image corresponding to the target scene.

[0124] Note that the above are only preferred embodiments of the present disclosure and the technical principles employed. Those skilled in the art will understand that the present disclosure is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure has been described in more detail through the above embodiments, the present disclosure is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the appended claims.

Claims

1. A depth camera calibration method, characterized in that: include: Control the depth camera to be calibrated to shoot the 3D calibration plate from multiple angles and / or multiple distances to obtain RGB images and depth maps; Extracting first feature point information of each calibration block in the RGB image and second feature point information of each calibration block in the depth image; wherein the first feature point information includes first feature point coordinates and first depth; and the second feature point information includes second feature point coordinates and second depth; Inputting the second feature point information into a set neural network model to obtain predicted feature point information; determining a loss function between the predicted feature point information and the first feature point information; Training a set neural network model based on the loss function to obtain a calibration model of the depth camera; wherein the calibration model of the depth camera refers to a model that corrects the coordinate information and depth information of each pixel point of the depth map based on the first feature point information; The step of extracting the first depth of each calibration block in the RGB image includes: Determine the distance between the optical center of the depth camera to be calibrated and the 3D calibration plate; The first depth is determined according to the distance and the height of the calibration block.

2. The method according to claim 1, characterized in that The 3D calibration plate is provided with a plurality of calibration blocks of different heights, and each calibration block is printed with a black and white pattern.

3. The method according to claim 2, characterized in that The black and white pattern is a three-dimensional pattern.

4. The method according to claim 2 or 3, characterized in that The first feature point and the second feature point are corner points of the black and white pattern.

5. The method according to claim 1, characterized in that After obtaining the calibration model of the depth camera, the method further includes: Control the calibrated depth camera to shoot the target scene and obtain the RGB image and initial depth map; Inputting the initial depth map into the calibration model to obtain a corrected depth map; The RGB image and the corrected depth image are aligned to obtain an RGBD image corresponding to the target scene.

6. A depth camera calibration device, characterized in that: include: The RGB image and depth map acquisition module is used to control the depth camera to be calibrated to shoot the 3D calibration plate from multiple angles and / or multiple distances to obtain the RGB image and depth map; A feature point information extraction module is configured to extract first feature point information of each calibration block in the RGB image and second feature point information of each calibration block in the depth image; wherein the first feature point information includes first feature point coordinates and first depth; and the second feature point information includes second feature point coordinates and second depth; a calibration model acquisition module, configured to input the second feature point information into a set neural network model to obtain predicted feature point information; determine a loss function between the predicted feature point information and the first feature point information; and train the set neural network model based on the loss function to obtain a calibration model for the depth camera; wherein the calibration model for the depth camera refers to a model for correcting the coordinate information and depth information of each pixel of the depth map based on the first feature point information; Wherein, the feature point information extraction module includes: The feature point information extraction unit of each calibration block is specifically used to determine the distance between the optical center of the depth camera to be calibrated and the 3D calibration plate; and determine the first depth according to the distance and the height of the calibration block.

7. An electronic device, characterized in that: The electronic device comprises: one or more processing devices; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the depth camera calibration method as described in any one of claims 1 to 5.

8. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the depth camera calibration method according to any one of claims 1 to 5 is implemented.

Images