A method, device, system and depth camera for obtaining true depth value

Abstract

The present application is applicable to the field of computer vision technology, and provides a method, device, system and depth camera for obtaining a depth truth value, wherein the method for obtaining a depth truth value comprises: obtaining a first depth image of a scene captured by a depth camera, and obtaining a second depth image of the same scene captured by a laser radar; converting the first depth image and the second depth image into first point cloud data and second point cloud data respectively; matching the first point cloud data and the second point cloud data to obtain a first transformation matrix; using the first transformation matrix, mapping the second point cloud data of the laser radar to the imaging plane of the depth camera to obtain the depth truth value. The embodiments of the present application do not require the equipment to be calibrated in advance, and do not need to consider changes in the external parameters of the equipment during use, and the image acquisition is flexible; in addition, a high-precision depth image of the depth camera can be easily generated.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method, device, system and depth camera for obtaining a true depth value. Background Art

[0002] In order to obtain better quality 3D information and solve some problems that are difficult to overcome in traditional 3D measurement methods, more and more 3D measurement methods are beginning to use deep learning algorithms to calculate depth. For example, deep learning is used to reconstruct the 3D information of a scene from a single RGB image or grayscale image, and deep learning is used to solve the depth error caused by multipath and scattering effects in indirect time of flight (iToF) measurement. Deep learning requires a large amount of data and ground truth as training samples. The number of training samples and the accuracy of the ground truth directly determine the final performance of the deep learning algorithm. Summary of the invention

[0003] In view of this, the embodiments of the present invention provide a method, device, system and depth camera for obtaining a true depth value, which can solve the technical problem of low accuracy of the true depth value.

[0004] In a first aspect, an embodiment of the present application provides a method for obtaining a depth truth value, comprising:

[0005] Acquire a first depth image of a scene captured by a depth camera, and acquire a second depth image of the same scene captured by a laser radar;

[0006] Converting the first depth image and the second depth image into first point cloud data and second point cloud data respectively;

[0007] Matching the first point cloud data and the second point cloud data to obtain a first transformation matrix;

[0008] The second point cloud data of the laser radar is mapped to the imaging plane of the depth camera using the first transformation matrix to obtain a true depth value.

[0009] The embodiment of the present application does not require prior calibration of the device, nor does it need to consider changes in device external parameters during use, and image acquisition is flexible; in addition, high-precision depth images of the depth camera can be easily generated.

[0010] As an implementation of the first aspect, the method for obtaining a true depth value further includes:

[0011] The first depth image and the true depth value are used as a group of training samples, multiple groups of training samples under different target scenes are obtained, and a neural network model is trained using the multiple groups of training samples to obtain a trained neural network model.

[0012] As an implementation of the first aspect, matching the first point cloud data and the second point cloud data to obtain a first transformation matrix includes:

[0013] Feature points are extracted from the first point cloud data and the second point cloud data respectively, and the feature points are described.

[0014] The correspondence between feature points with the same description is determined, and a second transformation matrix is ​​calculated.

[0015] Based on the second transformation matrix, iterate the correspondence between the first point cloud data and the second point cloud data, and obtain a third transformation matrix;

[0016] The third transformation matrix is ​​used to update the correspondence between the first point cloud data and the corresponding points of the second point cloud data, and the first transformation matrix is ​​iteratively solved.

[0017] As an implementation of the first aspect, using the third transformation matrix to update the correspondence between the first point cloud data and the corresponding points of the second point cloud data, and iteratively solving the first transformation matrix, includes:

[0018] Mapping the second point cloud data to the space where the first point cloud data is located using a third transformation matrix to obtain fourth point cloud data;

[0019] Find the corresponding points between the fourth point cloud data and the first point cloud data, and obtain the transformation relationship between the fourth point cloud data and the first point cloud data. Continuously iterate and loop to make the objective function value continuously decrease until it is less than a preset threshold, and obtain the final first transformation matrix.

[0020] In a second aspect, an embodiment of the present application provides a depth truth acquisition device, including:

[0021] An acquisition module, used to acquire a first depth image of a scene acquired by a depth camera, and acquire a second depth image of the same scene acquired by a laser radar;

[0022] A conversion module, used to convert the first depth image and the second depth image into first point cloud data and second point cloud data respectively;

[0023] A matching module, used for matching the first point cloud data and the second point cloud data to obtain a first transformation matrix;

[0024] The depth truth value calculation module is used to map the second point cloud data of the laser radar to the imaging plane of the depth camera using the first transformation matrix to obtain the depth truth value.

[0025] As an implementation of the second aspect, the depth truth value acquisition device also includes:

[0026] A training module is used to use the depth image and the true depth value as a group of training samples, obtain multiple groups of training samples in different scenarios, and use the multiple groups of training samples to train a neural network model to obtain a trained neural network model.

[0027] As an implementation manner of the second aspect, the matching module includes:

[0028] The extraction submodule is used to extract feature points from the first point cloud data and the second point cloud data respectively, and describe the feature points.

[0029] The calculation submodule is used to determine the corresponding relationship between feature points with the same description and calculate the second transformation matrix.

[0030] The iterative submodule is used to iterate the correspondence between the first point cloud data and the second point cloud data based on the second transformation matrix, and obtain a third transformation matrix.

[0031] The solving submodule is used to update the corresponding relationship between the first point cloud data and the corresponding points of the second point cloud data by using the third transformation matrix, and iteratively solve the first transformation matrix.

[0032] As an implementation of the second aspect, the solution submodule is specifically used to:

[0033] Mapping the second point cloud data to the space where the first point cloud data is located using a third transformation matrix to obtain fourth point cloud data;

[0034] Find the corresponding points between the fourth point cloud data and the first point cloud data, and obtain the transformation relationship between the fourth point cloud data and the first point cloud data. Continuously iterate and loop to make the objective function value continuously decrease until it is less than a preset threshold, and obtain the final first transformation matrix.

[0035] In a third aspect, an embodiment of the present application provides a depth truth value acquisition system, including a depth camera and a laser radar, and the depth truth value acquisition system also includes a depth truth value acquisition device as described in the second aspect or any implementation method of the second aspect.

[0036] In a fourth aspect, an embodiment of the present application provides a depth camera, including a projection module, a collection module and a processing module.

[0037] The projection module is used to project a pulse light beam to a target area;

[0038] The acquisition module is used to receive the pulse light beam reflected back from the target area and generate a target image;

[0039] The processing module includes a trained training unit, which is used to calculate the depth value of the target image and generate a third depth image, and then use the third depth image and the training unit to obtain the true depth value of the target area;

[0040] Among them, the training unit is a neural network model that is pre-trained using the first depth image obtained by the depth truth value acquisition method described in the first aspect or any implementation method of the first aspect as a training sample.

[0041] In a fifth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the depth truth acquisition method as described in any implementation method of the first aspect or the first direction are implemented.

[0042] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps of the depth truth acquisition method described in the first aspect or any implementation method of the first aspect.

[0043] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 It is a structural diagram of a depth truth acquisition system provided by an embodiment of the present application;

[0046] Figure 2 is a structural diagram of a processing unit provided in one embodiment of the present application;

[0047] Figure 3 It is a schematic diagram of an implementation flow of a method for obtaining a depth truth value provided in an embodiment of the present application;

[0048] Figure 4 It is a schematic diagram of a matching process of a point cloud of a depth camera and a point cloud of a laser radar provided in an embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of a specific implementation flow of step S330 in a method for obtaining a depth truth value provided in an embodiment of the present application;

[0050] Figure 6 is a schematic diagram of a depth truth acquisition device provided by an embodiment of the present application;

[0051] Figure 7 is a schematic diagram of a matching module in a depth truth acquisition device provided in an embodiment of the present application;

[0052] Figure 8 It is a schematic diagram of a depth truth acquisition device provided in an embodiment of the present application.

[0053] Fig. 9 is a schematic diagram of a depth camera provided in one embodiment of the present application. DETAILED DESCRIPTION

[0054] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0055] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0056] "One embodiment" or "some embodiments" described in the specification of this application means that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the sentences "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0057] In addition, in the description of the present application, "plurality" means two or more. The terms "first", "second", "third", and "fourth" are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0058] It should also be understood that, unless otherwise clearly specified or limited, the term "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0059] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0060] Figure 1 This is a schematic diagram of the structure of a depth truth acquisition system provided in one embodiment of the present application. Figure 1 As shown, the depth truth acquisition system includes a depth camera 11, a laser radar 12, and a processing unit 13. The depth camera 11 and the laser radar 12 can both exchange data with the processing unit 13.

[0061] The depth camera 11 is used to collect a first depth image of the scene. The laser radar 12 is used to collect a second depth image of the scene. The processing unit 13 is used to obtain the true depth value of the scene under the depth camera's perspective according to the first depth image and the second depth image.

[0062] In some embodiments, the depth camera 11 includes a projection module and an infrared receiving module, and the laser radar 12 includes a transmitting end and a receiving end. The depth camera 11 and the laser radar 12 are placed close to each other, that is, the infrared receiving module of the depth camera 11 is placed next to the receiving end of the laser radar 12, and the optical axis of the infrared receiving module of the depth camera 11 is parallel to the optical axis of the receiving end of the laser radar 12, and the imaging plane pixel rows of the depth camera 11 are parallel to the imaging plane pixel rows of the laser radar 12.

[0063] The depth camera 11 is an imaging device that can obtain depth information of a scene. In some embodiments, the depth camera 11 may be an iToF camera. In some other embodiments, the depth camera 11 may be a direct time of flight (dToF) camera. In some other embodiments, the depth camera 11 may also be a depth camera based on structured light or laser scanning, and the embodiments of the present application do not limit the type of the depth camera.

[0064] As a non-limiting example, the depth camera is an iToF camera as an example for explanation. The iToF camera includes a projection module and an infrared receiving module. Among them, the projection module may include a light source and an optical component. The light source may be an edge emitting laser (EEL), a vertical cavity surface emitting laser (VCSEL) and other light sources, or a light source array composed of multiple light sources. The light beam emitted by the light source may be visible light, infrared light, ultraviolet light, etc. The optical component may include a diffraction optical element, etc. The light beam emitted by the light source can form a uniform, random or specially designed intensity distribution projection pattern on the reference plane after passing through the optical component. The infrared receiving module is an image sensor. The image sensor may be an image sensor composed of a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), an avalanche diode (AD), a single photon avalanche diode (SPAD), etc., and this application does not limit this.

[0065] In some embodiments, the laser radar 12 may be an array laser radar. As a non-limiting example, the array laser radar includes a laser transmitter and a photoelectric detection receiving end. The laser transmitter is used to emit a light source. As a non-limiting implementation, the laser transmitter may be a semiconductor laser. More specifically, the laser transmitter may be a VCSEL array light source chip formed by generating multiple VCSEL light sources on a single semiconductor substrate. The VCSEL array light source chip may emit a pulse beam outward at a certain frequency (or referred to as a pulse period), and the pulse beam is projected onto the scene to form an illumination spot (i.e., a light spot). The frequency of the pulse beam can be set according to the measurement distance. The photoelectric detection receiving end is used to receive the echo signal of the pulse beam, and to determine the depth information of the scene by capturing the light intensity information of the echo signal.

[0066] It should be noted that in some other examples, the laser transmitter may also be other types of lasers such as a solid-state laser, and the photoelectric detection receiving end may be a planar array light intensity collection device such as a CCD, and this application does not impose any restrictions on the two.

[0067] exist Figure 1In the illustrated embodiment, the processing unit 13 is a computer. In other embodiments, the processing unit 14 may include an electronic device such as a vehicle-mounted terminal, a robot, a mobile phone, a tablet computer, a laptop computer, a netbook, a personal digital assistant (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of the processing unit 13.

[0068] In some embodiments of the present application, Figure 2 As shown, the processing unit may include one or more processors 20 ( Figure 2 Only one is shown in the figure), a memory 21 and a computer program 22 stored in the memory 21 and executable on one or more processors 20, for example, a program for acquiring a depth truth value. When one or more processors 20 execute the computer program 22, the various steps in the depth truth value acquisition method embodiment described later can be implemented. Alternatively, when one or more processors 20 execute the computer program 22, the functions of the various modules / units in the depth truth value acquisition device embodiments described later can be implemented, which are not limited here.

[0069] Those skilled in the art will understand that Figure 2 This is only an example of a processing unit and does not constitute a limitation on the processing unit. The processing unit may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the processing unit may also include input and output devices, network access devices, buses, etc.

[0070] In one embodiment, the processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0071] In one embodiment, the memory 21 may be an internal storage unit of the processing unit, such as a hard disk or memory of the processing unit. The memory 21 may also be an external storage device of the processing unit, such as a plug-in hard disk equipped on the processing unit, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 21 may also include both an internal storage unit and an external storage device of the processing unit. The memory 21 is used to store computer programs and other programs and data required by the processing unit. The memory 21 may also be used to temporarily store data that has been output or is to be output.

[0072] An embodiment of the present application also provides another preferred embodiment of the processing unit 14. In this embodiment, the processing unit includes one or more processors, and the one or more processors are used to execute the following program modules stored in the memory: an acquisition module, a conversion module, a matching module, and a depth truth calculation module.

[0073] Among them, the acquisition module is used to acquire a first depth image of a scene captured by a depth camera, and acquire a second depth image of the same scene captured by a laser radar.

[0074] The conversion module is used to convert the first depth image and the second depth image into the first point cloud data and the second point cloud data respectively.

[0075] The matching module is used to match the first point cloud data and the second point cloud data to obtain a first transformation matrix.

[0076] The depth truth value calculation module is used to map the second point cloud data of the laser radar to the imaging plane of the depth camera using the first transformation matrix to obtain the depth truth value.

[0077] In some embodiments, the processing unit can be independent of the depth camera and the lidar, see Figure 1 Embodiment shown. In some other embodiments, the processing unit may be integrated into at least one of the binocular camera and the target camera. That is, in some other embodiments, one or more processors of the depth camera and / or the laser radar itself may implement one or more steps in the depth truth value acquisition method embodiment described later when executing a computer program stored in a memory. Alternatively, one or more processors of the depth camera and / or the laser radar itself may implement the functions of one or more modules / units in each depth truth value acquisition device embodiment described later when executing a computer program stored in a memory.

[0078] See also Figure 3 , Figure 31 is a schematic diagram of the implementation flow of a method for obtaining a depth truth value provided in an embodiment of the present application. The method for obtaining a depth truth value in this embodiment can be executed by a processing unit, which includes but is not limited to a computer, a depth camera or a laser radar. As an example and not a limitation, the method for obtaining a depth truth value can be applied to Figure 1 The depth truth value acquisition system shown in the figure, the depth truth value acquisition method can be Figure 1 or Figure 2 The processing unit in the embodiment shown in the figure executes. The method for obtaining the true depth value in this embodiment is applicable to the situation where the true depth value of the scene needs to be obtained. Figure 3 The method for obtaining the true depth value shown may include: steps S310 to S340.

[0079] S310, obtaining a first depth image of a scene captured by a depth camera, and obtaining a second depth image of the same scene captured by a laser radar.

[0080] The processing unit obtains a first depth image and a second depth image of the same scene respectively collected by a depth camera and a laser radar, wherein the depth camera is used to collect the first depth image of the scene, and the laser radar is used to collect the second depth image of the same scene.

[0081] In some embodiments, in order to avoid mutual interference of light sources between the depth camera and the laser radar in the process of acquiring the depth image, the depth camera and the laser radar can collect depth images of the same scene in sequence. As a non-limiting example, the processing unit first turns on the depth camera to collect the first depth image of the scene; then turns off the depth camera and turns on the laser radar to control the laser radar to collect the second depth image of the same scene. As another non-limiting example, the depth camera and the laser radar can be manually operated to collect the first depth image and the second depth image of the same scene in sequence. It should be understood that the order can be swapped, and the present application does not limit the order of collecting the first depth image and the second depth image.

[0082] The first depth image captured by the depth camera and the second depth image captured by the lidar are transmitted to the processing unit for subsequent processing.

[0083] In some embodiments, the processing unit includes a communication module and / or an input / output device, and can obtain the first depth image and the second depth image from the depth camera and the laser radar respectively through the communication module and / or the input / output device. Figure 1In the embodiment shown, in some implementations, the computer receives the first depth image and the second depth image actively transmitted by the depth camera and the laser radar, that is, the depth camera and the laser radar can actively send the collected first depth image and the second depth image to the computing unit after collecting the first depth image and the second depth image respectively. In some other implementations, the processing unit can actively obtain the first depth image and the second depth image collected by the depth camera and the laser radar.

[0084] In some other embodiments, the processing unit includes a processor, and the processing unit can obtain a first depth image of a scene captured by a depth camera and a second depth image of the same scene captured by a laser radar from a memory (e.g., an external storage device or an internal storage unit) coupled to the processor. The first depth image of the scene captured by the depth camera and the second depth image of the same scene captured by the laser radar are stored in the memory in advance, and when it is necessary to obtain the true depth value of the scene, the processing unit calls the first depth image and the second depth image stored in advance from the memory.

[0085] S320: Convert the first depth image and the second depth image into first point cloud data and second point cloud data respectively.

[0086] Specifically, the processing unit converts a first depth image captured by the depth camera into first point cloud data, and converts a second depth image captured by the laser radar into second point cloud data.

[0087] As a non-limiting example, a calculation method for converting the first depth image into the first point cloud data is:

[0088]

[0089] Among them, (x s ,y s ,z s ) is the three-dimensional coordinate of the first point cloud in the depth camera coordinate system, z is the depth of each pixel, (u,v) is the pixel coordinate, (u0,v0) is the image principal point coordinate, dx and dy are the physical sizes of the sensor pixels of the depth camera in two directions, and f' is the focal length (in millimeters).

[0090] Similarly, based on the above calculation method, the second depth image collected by the laser radar can be converted into second point cloud data.

[0091] S330, matching the first point cloud data and the second point cloud data to obtain a first transformation matrix.

[0092] Matching the first point cloud data and the second point cloud data, that is, registering the first point cloud data and the second point cloud data. The registration process of the point cloud of the depth camera and the point cloud of the laser radar can be referred to Figure 4shown.

[0093] In some embodiments, Figure 5 As shown, step S330 may include steps S331 to S334.

[0094] S331, extracting feature points from the first point cloud data and the second point cloud data respectively, and describing the feature points.

[0095] Feature points are extracted from the first point cloud data and the second point cloud data, where the feature points include but are not limited to sharp edges, smooth edges, ridges or valleys, and sharp points.

[0096] The description of the feature point may be a description of the geometric properties of the feature point, which include but are not limited to normal vectors and curvature.

[0097] S332, determining the correspondence between feature points with the same description, and calculating a second transformation matrix.

[0098] After describing the feature points, the feature points with the same description (e.g., geometric attributes) are regarded as corresponding points in the correspondence relationship between the first point cloud data and the second point cloud data. It should be understood that two feature points with the same description (e.g., geometric attributes) in the first point cloud data and the second point cloud data are a pair of corresponding points.

[0099] In some embodiments, the first point cloud data is used as reference point cloud data, feature points with the same description (such as geometric attributes) in the first point cloud data and the second point cloud data are obtained, and the feature points with the same description in the second point cloud data are mapped to the first point cloud data to obtain a second transformation matrix.

[0100] It should be noted that, by extracting feature points in step S331 and retaining points with the same description and significant features, the number of points that need to be considered for registration can be reduced, the amount of calculation can be reduced, and the calculation efficiency can be improved.

[0101] S333: based on the second transformation matrix, iterate the correspondence between the first point cloud data and the second point cloud data, and obtain a third transformation matrix.

[0102] In some embodiments, an iterative closest point (ICP) algorithm may be used to calculate a third transformation matrix between the first point cloud data and the second point cloud data.

[0103] More specifically, first, all points in the second point cloud data are mapped to the space where the first point cloud data is located through the second transformation matrix to obtain new point cloud data, which is the third point cloud data.

[0104] The core of the ICP algorithm is to minimize the objective function. Taking the first point cloud data as the reference point cloud data, the objective function is:

[0105]

[0106] Among them, R is the rotation matrix, T is the translation matrix, and p t i and p s i is the i-th pair of corresponding points in the first point cloud data and the third point cloud data, N p Characterizes that there is a total of N between the first point cloud data and the third point cloud data p For corresponding points. Preferably, the corresponding points are the nearest points in the third point cloud data and the first point cloud data, that is, the Euclidean distance between the third point cloud data and the point cloud in the first point cloud data is calculated, and if the Euclidean distance is within a preset distance threshold, the two point clouds are corresponding points. It should be noted that the preset distance threshold is usually set according to the accuracy requirements of the final result, and this application is not limited to this.

[0107] Next, the first transformation matrix [R, T] is calculated according to the objective function. R and T have only 6 degrees of freedom and the number of corresponding points is huge. Therefore, the least squares method can be used to solve the optimal solution of the third transformation matrix.

[0108] It should be noted that the method for solving the third transformation matrix is ​​not limited to the ICP algorithm, but may also be a singular value decomposition (SVD) algorithm, which is not limited in the present application.

[0109] S334: using the third transformation matrix, updating the correspondence between the first point cloud data and the corresponding points of the second point cloud data, and iteratively solving the first transformation matrix.

[0110] In some embodiments, the third transformation matrix is ​​a new transformation matrix. With the first point cloud data as the reference point cloud data, the second point cloud data is mapped to the space where the first point cloud data is located using the third transformation matrix, and a new point cloud data, i.e., the fourth point cloud data, can be obtained. After obtaining the fourth point cloud data, step S333 is looped, and the iterative closest point algorithm is used to find the corresponding points of the fourth point cloud data and the first point cloud data, and then the transformation relationship between the fourth point cloud data and the first point cloud data is obtained, and the iteration cycle is continuously performed to continuously reduce the objective function value until it is less than the preset threshold value, and the final transformation relationship, i.e., the first transformation matrix, is obtained.

[0111] S340, using the first transformation matrix, mapping the second point cloud data of the laser radar to the imaging plane of the depth camera to obtain a true depth value.

[0112] In some embodiments, the second point cloud data of the laser radar is mapped to the imaging plane of the depth camera using the first transformation matrix. According to the second point cloud data projected to the imaging plane of the depth camera, the depth information of the scene acquired by the depth camera is calculated, and the depth information is used as the true depth value of the depth camera.

[0113] As a non-limiting example, a method for calculating the true depth value of the scene acquired by the depth camera according to the second point cloud data projected onto the imaging plane of the depth camera is:

[0114]

[0115] Among them, (x s ,y s ,z s ) is the three-dimensional coordinate of the second point cloud, K is the internal parameter of the depth camera, [R, T] is the first transformation matrix, (u, v) is the pixel coordinate corresponding to the second point cloud data after being mapped to the depth camera, z c is the true depth value of the point cloud in the depth camera coordinate system.

[0116] The embodiments of the present application do not require prior calibration of the device, nor do they need to consider changes in device external parameters during use, and image acquisition is flexible; in addition, a high-precision depth map that is pixel-by-pixel aligned with the imaging device in the depth camera can be easily generated.

[0117] It should be noted that in some other embodiments, since the point cloud of the laser radar is relatively sparse, the depth truth value obtained by the depth camera can be interpolated to obtain complete depth truth information. The interpolation method includes the nearest neighbor interpolation method, the bilinear interpolation method, or the edge-guided interpolation method, which is not limited here. As a non-limiting example, the depth truth value z can be interpolated according to the pixel coordinates. c Interpolation is performed to obtain the true depth value of the scene from the depth camera's perspective.

[0118] In the aforementioned embodiment, through steps S310 to S340, a first depth image of the scene and a depth truth value corresponding to the first depth image are obtained. Further, in some other embodiments, the first depth image and the corresponding depth truth value can be used as a set of training data. For multiple different scenes, multiple sets of training data can be obtained, and the multiple sets of training data constitute training samples. The neural network model is trained using the training samples to obtain the weight parameters of the neural network model and obtain a trained neural network model. The trained neural network model can be deployed in electronic devices, such as computers, mobile phones, or cameras. When used, the electronic device can obtain a higher-precision depth truth value of a scene or area based on a depth image of the scene or area obtained by a camera or camera, using the trained neural network model.

[0119] The embodiment of the present application uses a depth camera to obtain a first depth image of a scene, and uses it as an input of a neural network model, and obtains the weight parameters of the neural network model by learning the depth truth value of the depth camera. It should be understood that after the above steps S310 to S340, the depth camera can obtain the depth truth value of different scenes based on the laser radar, use the depth camera to obtain the first depth image of different scenes, and use the neural network model to learn the depth truth value of different scenes. The weight parameters of the neural network model can be updated and iterated to ensure that the depth image subsequently obtained by the depth camera can be input into the neural network model to obtain a higher precision depth image.

[0120] The neural network model can be, for example, a convolutional neural network model, a fully connected neural network model, etc., which is not limited in this application.

[0121] Deep learning requires a large amount of training data (such as images) and ground truth as training samples. The number of training samples and the accuracy of the ground truth directly determine the final performance of the deep learning algorithm. The embodiments of the present application can accurately and efficiently obtain the training samples required for end-to-end deep learning neural networks through simple methods and systems.

[0122] Corresponding to the above-mentioned method for obtaining a true depth value, an embodiment of the present application further provides a device for obtaining a true depth value. For details not described in detail in the device for obtaining a true depth value, please refer to the description of the above-mentioned method.

[0123] See also Figure 6 , Figure 6 is a schematic block diagram of a depth truth value acquisition device provided by an embodiment of the present invention. The depth truth value acquisition device can be configured in an electronic device. As an example, the depth truth value acquisition device is configured in Figure 1 or Figure 2 The processing unit shown is connected to the depth camera and the lidar.

[0124] The apparatus for acquiring a true depth value comprises: an acquisition module 61 , a conversion module 62 , a matching module 63 , and a true depth value calculation module 64 .

[0125] The acquisition module 61 is used to acquire a first depth image of a scene captured by a depth camera, and acquire a second depth image of the same scene captured by a laser radar.

[0126] The conversion module 62 is used to convert the first depth image and the second depth image into first point cloud data and second point cloud data respectively.

[0127] The matching module 63 is used to match the first point cloud data and the second point cloud data to obtain a first transformation matrix.

[0128] The depth truth calculation module 64 is used to map the second point cloud data of the laser radar to the imaging plane of the depth camera using the first transformation matrix to obtain the depth truth.

[0129] Optional, such as Figure 7 As shown, the apparatus for acquiring the true depth value further includes a training module 65. The training module 65 is configured to use the first depth image and the true depth value as a set of training samples, acquire multiple sets of training samples under different target scenes, and train the neural network model using the multiple sets of training samples to obtain a trained neural network model.

[0130] Optional, such as Figure 8 As shown, the matching module 63 includes: an extraction submodule 631 , a calculation submodule 632 , an iteration submodule 633 , and a solution submodule 634 .

[0131] The extraction submodule 631 is used to extract feature points from the first point cloud data and the second point cloud data respectively, and describe the feature points.

[0132] The calculation submodule 632 is used to determine the correspondence between feature points with the same description and calculate the second transformation matrix.

[0133] The iterative submodule 633 is used to iterate the correspondence between the first point cloud data and the second point cloud data based on the second transformation matrix, and obtain a third transformation matrix.

[0134] The solving submodule 634 is used to update the correspondence between the first point cloud data and the corresponding points of the second point cloud data using the third transformation matrix, and iteratively solve the first transformation matrix.

[0135] Fig. 9 A schematic diagram of the structure of a depth camera provided according to the present invention, the depth camera includes a projection module, an acquisition module and a processing module, wherein the projection module is used to project a pulse light beam to a target area; the acquisition module is used to acquire the pulse light beam reflected back from the target area and generate a target image; the processing module includes a training unit, the processing module is used to calculate the depth value of the target image and generate a third depth image, and then use the third depth image and the training unit to obtain the depth truth value of the target area, wherein the training unit is a trained neural network model, more specifically, the first depth image obtained in advance using the above-mentioned depth truth value acquisition method is used as a training sample to train the neural network model to obtain a trained neural network model.

[0136] In some embodiments, calculating the depth value of the target image and generating the third depth image may include at least one of the following three methods:

[0137] Method ①: The projection module projects a structured light beam to the target area, and the acquisition module receives the beam reflected from the target area and forms an electrical signal. The electrical signal is transmitted to the processing module, which processes the electrical signal and calculates the intensity information reflecting the beam to form a structured light pattern. Finally, matching calculation or triangulation calculation is performed based on the structured light pattern to obtain the depth value.

[0138] Method ②: The projection module projects an infrared beam to the target area, and the acquisition module receives the beam reflected from the target area and forms an electrical signal. The electrical signal is transmitted to the processing module, which processes the electrical signal to calculate the phase difference, and indirectly calculates the flight time taken by the beam from the projection module to the acquisition module based on the phase difference, and further calculates the depth value based on the flight time. It should be understood that the infrared beam may include pulse type and continuous wave type, which is not limited here.

[0139] Method ③: The projection module projects an infrared pulse beam to the area, and the acquisition module receives the beam reflected from the target area and forms an electrical signal. The electrical signal is transmitted to the processing module, which counts the electrical signal to obtain a waveform histogram, and directly calculates the flight time from the projection module to the acquisition module based on the histogram, and further calculates the depth value based on the flight time.

[0140] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0141] An embodiment of the present application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned depth truth value acquisition method embodiments when executing the computer program.

[0142] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned depth truth value acquisition method embodiments can be implemented.

[0143] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in any of the above-mentioned depth truth acquisition method embodiments.

[0144] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0145] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0146] In the embodiments provided by the present invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0149] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0150] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for obtaining a depth truth value, characterized in that: include: Acquire a first depth image of a scene captured by a depth camera, and acquire a second depth image of the same scene captured by a laser radar, wherein the depth camera is an iToF camera and the laser radar is an array laser radar; Converting the first depth image and the second depth image into first point cloud data and second point cloud data respectively; Matching the first point cloud data with the second point cloud data to obtain a first transformation matrix includes: Extracting feature points from the first point cloud data and the second point cloud data respectively, and describing the feature points; Determine the correspondence between feature points with the same description and calculate a second transformation matrix; Based on the second transformation matrix, iterate the correspondence between the first point cloud data and the second point cloud data, and obtain a third transformation matrix; Using the third transformation matrix, updating the correspondence between the first point cloud data and the corresponding points of the second point cloud data, and iteratively solving the first transformation matrix; Mapping the second point cloud data of the laser radar to the imaging plane of the depth camera using the first transformation matrix to obtain a true depth value; The method further includes: interpolating the true depth value obtained by the depth camera to obtain complete true depth value information.

2. The method for obtaining a true depth value according to claim 1, wherein: Also includes: The first depth image and the true depth value are used as a group of training samples, multiple groups of training samples under different target scenes are obtained, and a neural network model is trained using the multiple groups of training samples to obtain a trained neural network model.

3. The method for obtaining a true depth value according to claim 1, wherein: The method of using the third transformation matrix to update the correspondence between the first point cloud data and the corresponding points of the second point cloud data, and iteratively solving the first transformation matrix, includes: Mapping the second point cloud data to the space where the first point cloud data is located using a third transformation matrix to obtain fourth point cloud data; Find the corresponding points between the fourth point cloud data and the first point cloud data, and obtain the transformation relationship between the fourth point cloud data and the first point cloud data. Continuously iterate and loop to make the objective function value continuously decrease until it is less than a preset threshold, and obtain the final first transformation matrix.

4. A device for obtaining a depth truth value, characterized in that: include: An acquisition module is used to acquire a first depth image of a scene captured by a depth camera, and acquire a second depth image of the same scene captured by a laser radar, wherein the depth camera is an iToF camera, and the laser radar is an array laser radar; A conversion module, used to convert the first depth image and the second depth image into first point cloud data and second point cloud data respectively; A matching module, used for matching the first point cloud data and the second point cloud data to obtain a first transformation matrix; A depth truth value calculation module, used to map the second point cloud data of the laser radar to the imaging plane of the depth camera using the first transformation matrix to obtain a depth truth value; The matching module comprises: An extraction submodule, used to extract feature points from the first point cloud data and the second point cloud data respectively, and describe the feature points; A calculation submodule, used to determine the correspondence between feature points with the same description and calculate a second transformation matrix; An iterator module, used for iterating the correspondence between the first point cloud data and the second point cloud data based on the second transformation matrix, and obtaining a third transformation matrix; A solving submodule, used to update the corresponding relationship between the first point cloud data and the corresponding points of the second point cloud data using the third transformation matrix, and iteratively solve the first transformation matrix; In addition, the true depth value obtained by the depth camera is interpolated to obtain complete true depth value information.

5. The apparatus for obtaining a depth truth value according to claim 4, wherein: Also includes: A training module is used to use the first depth image and the true depth value as a group of training samples, obtain multiple groups of training samples in different scenarios, and use the multiple groups of training samples to train a neural network model to obtain a trained neural network model.

6. A system for acquiring a depth truth value, characterized in that: The system comprises a depth camera and a laser radar, and the depth truth value acquisition system also comprises a depth truth value acquisition device as described in claim 4 or 5.

7. A depth camera, characterized in that: It includes projection module, acquisition module and processing module. The projection module is used to project a pulse light beam to a target area; The acquisition module is used to receive the pulse light beam reflected back from the target area and generate a target image; The processing module comprises a trained training unit, the processing module is used to calculate the depth value of the target image and generate a third depth image, and then use the third depth image and the training unit to obtain the true depth value of the target area; Wherein, the training unit is a neural network model that is pre-trained using a first depth image obtained by the depth truth acquisition method described in any one of claims 1 to 3 as a training sample.

8. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the depth truth acquisition method according to any one of claims 1 to 3 are implemented.

Images