Depth information reconstruction method, device, equipment, storage medium and computer program product

Abstract

The present application relates to the field of structured light application technology, and discloses a method, device, equipment, storage medium and computer program product for reconstructing depth information, including: calculating the disparity information of the image to be measured based on a preset structured light array; determining the initial depth information of the image to be measured based on the disparity information, and reconstructing the disparity information based on the initial depth information to obtain real disparity information; establishing a depth information joint reconstruction calculation model based on the real disparity information and the characteristic calibration distance of the structured light array; reconstructing the initial depth information through the depth information joint reconstruction calculation model, and outputting the target depth information. By utilizing the diversity of the characteristic calibration distance, more flexible depth reference information is provided for disparity calculation, and more constraints are fully utilized when calculating the depth through the depth information joint reconstruction model, effectively reducing the cumulative effect of errors, and improving the depth measurement accuracy, thereby enhancing the accuracy and reliability of the depth measurement.

Description

Technical Field

[0001] The present application relates to the field of structured light application technology, and in particular to a depth information reconstruction method, device, equipment, storage medium and computer program product. Background Art

[0002] The structured light projected by traditional structured light systems is usually static and fixed, and it is impossible to finely control the structured light. It is difficult to fully utilize the characteristics of structured light to improve the accuracy of depth measurement, resulting in errors in parallax calculations. Summary of the invention

[0003] The main purpose of the present application is to provide a depth information reconstruction method, device, equipment, storage medium and computer program product, aiming to solve the technical problem that the structured light projected by the traditional structured light system is statically fixed, resulting in errors in parallax calculation.

[0004] To achieve the above objectives, the present application proposes a depth information reconstruction method, which includes:

[0005] Calculating the parallax information of the image to be measured based on a preset structured light array;

[0006] Determining initial depth information of the image to be tested according to the disparity information, and reconstructing the disparity information based on the initial depth information to obtain real disparity information;

[0007] Establishing a depth information joint reconstruction calculation model according to the real disparity information and the characteristic calibration distance of the structured light array;

[0008] The initial depth information is reconstructed through the depth information joint reconstruction calculation model to output target depth information.

[0009] Optionally, the step of calculating the disparity information of the image to be measured based on a preset structured light array includes:

[0010] The structured light array is projected onto the image to be measured and a plane at a specific distance to generate a deformed image and a reference image;

[0011] Selecting a speckle in the deformed image as a center to be measured, and selecting a speckle image block based on the center to be measured;

[0012] A matching image block corresponding to the speckle image block is identified from the reference image according to an image matching algorithm, and an offset between the speckle image block and the matching image block is calculated to obtain disparity information.

[0013] Optionally, before the step of projecting the structured light array onto the image to be measured and the plane at a specific distance to generate the deformed image and the reference image, the method further includes:

[0014] Acquiring measurement information of the image to be measured;

[0015] The parameters of the structured light array generation module are adjusted according to the measurement information, and a structured light array with a characteristic calibration distance is generated based on the adjusted structured light array generation module.

[0016] Optionally, the step of determining initial depth information of the image to be measured according to the disparity information, and reconstructing the disparity information based on the initial depth information to obtain real disparity information includes:

[0017] Calculating the depth value of each pixel in the image to be tested according to the disparity information to obtain initial depth information;

[0018] Acquire a temperature drift error, and establish a depth information calculation model through the initial depth information, the parallax information, and the temperature drift error;

[0019] The depth information calculation model is solved by using regularization to obtain real disparity information.

[0020] Optionally, the step of establishing a depth information joint reconstruction calculation model according to the real disparity information and the characteristic calibration distance of the structured light array includes:

[0021] Calculating a number of characteristic calibration distances of the structured light array, and determining position data of scattered spots in the structured light array based on the number;

[0022] A depth information joint reconstruction calculation model is established according to the position data and the real disparity information.

[0023] Optionally, the step of reconstructing the initial depth information by using the depth information joint reconstruction calculation model and outputting target depth information includes:

[0024] Determining a joint optimization problem of the initial depth information according to the depth information joint reconstruction calculation model;

[0025] The joint optimization problem is solved using a regularization algorithm, and target depth information is output.

[0026] In addition, to achieve the above-mentioned purpose, the present application also proposes a depth information reconstruction device, the depth information reconstruction device comprising:

[0027] A disparity calculation module, used for calculating the disparity information of the image to be measured based on a preset structured light array;

[0028] A disparity reconstruction module, used to determine the initial depth information of the image to be measured according to the disparity information, and reconstruct the disparity information based on the initial depth information to obtain real disparity information;

[0029] A model building module, used to establish a depth information joint reconstruction calculation model according to the real disparity information and the characteristic calibration distance of the structured light array;

[0030] The information reconstruction module is used to reconstruct the initial depth information through the depth information joint reconstruction calculation model and output target depth information.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a depth information reconstruction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the depth information reconstruction method described above.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the depth information reconstruction method described above are implemented.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the depth information reconstruction method described above are implemented.

[0034] The present application discloses calculating the disparity information of the image to be measured based on a preset structured light array; determining the initial depth information of the image to be measured based on the disparity information, and reconstructing the disparity information based on the initial depth information to obtain real disparity information; establishing a depth information joint reconstruction calculation model based on the real disparity information and the characteristic calibration distance of the structured light array; reconstructing the initial depth information through the depth information joint reconstruction calculation model, and outputting the target depth information. By utilizing the diversity of the characteristic calibration distance, more flexible depth reference information is provided for disparity calculation, and more constraints are fully utilized when calculating the depth through the depth information joint reconstruction model, effectively reducing the cumulative effect of errors, and improving the depth measurement accuracy, thereby enhancing the accuracy and reliability of the depth measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0037] Figure 1 This is a flowchart of the first embodiment of the depth information reconstruction method of the present application;

[0038] Figure 2 A module diagram of the optoelectronic device for this application;

[0039] Figure 3 This is a flow chart of a second embodiment of the depth information reconstruction method of the present application;

[0040] Figure 4 A scanning electron microscope image of a metasurface of the metasurface array of the present application;

[0041] Figure 5 Schematic diagram of a coded structured light array with characteristic calibration distance generated for this application;

[0042] Figure 6 This is a schematic diagram of the flow chart of the third embodiment of the depth information reconstruction method of the present application;

[0043] Figure 7 This is a flowchart of a fourth embodiment of the depth information reconstruction method of the present application;

[0044] Figure 8 Schematic diagram of the structured light array generated for a single VCSEL aperture;

[0045] Fig. 9 Schematic diagram of a coded structured light array with characteristic calibration distance generated for four VCSEL apertures;

[0046] Fig.10 Schematic diagram of the coded structured light array with characteristic calibration distance generated for 3×3 VCSEL light holes;

[0047] Fig.11 Schematic diagram of the calculation of the characteristic calibration distance generated for the N×M VCSEL array light hole;

[0048] Fig.12 This is a schematic diagram of the module structure of the depth information reconstruction device according to an embodiment of the present application;

[0049] Fig.13 Schematic diagram of the device structure of the hardware operating environment involved in the depth information reconstruction method in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the embodiment of the present application is: calculate the disparity information of the image to be measured based on a preset structured light array; determine the initial depth information of the image to be measured based on the disparity information, and reconstruct the disparity information based on the initial depth information to obtain real disparity information; establish a depth information joint reconstruction calculation model based on the real disparity information and the characteristic calibration distance of the structured light array; reconstruct the initial depth information through the depth information joint reconstruction calculation model, and output target depth information.

[0054] As the era of spatial computing develops in depth, three-dimensional depth perception is attracting more and more interest and attention due to its important applications in machine vision and artificial intelligence. At present, non-contact optical three-dimensional depth measurement methods are mainly divided into structured light projection, time-of-flight method and stereoscopic vision. Among them, the three-dimensional perception technology based on structured light has the characteristics of high spatial resolution, miniaturization, strong anti-interference ability, etc., and is widely used in machine vision, face recognition, human-computer interaction, somatosensory games, bionic robots and other fields. A common structured light depth perception system is mainly composed of VCSEL (Vertical-Cavity Surface-Emitting Laser) array, collimating lens, DOE (Diffractive Optical Element) and infrared receiver. Among them, the VCSEL projector emits near-infrared light of a specific wavelength, and the light beam collimated by the lens is projected onto the DOE to form a specific structured light pattern, thereby marking the three-dimensional space of the target object to be measured, and then the infrared camera receives the reflected structured light speckle pattern, and realizes the three-dimensional shape detection of the object to be measured by calculating the deformation of the structured light speckle. However, the structured light projected by this method is usually static and fixed, resulting in low flexibility and inability to perform micron-level fine control of the structured light. The depth detection accuracy using it is generally 1mm.

[0055] Therefore, the present application provides a method for joint reconstruction of depth information based on feature calibration distance and an optoelectronic device having the method. By utilizing the diversity of speckle feature calibration distances, more flexible depth reference information is provided for parallax calculation, and the joint reconstruction model of depth information can fully utilize more constraints when calculating depth, effectively reduce the cumulative effect of errors, and significantly improve the depth measurement accuracy, thereby enhancing the accuracy and reliability of depth measurement; and further expand the practical application of high-precision structured light systems in the fields of artificial intelligence, face recognition, machine vision, etc., and promote the further development of 3D imaging-based consumer electronic product technology.

[0056] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, light source projection and program running functions, such as a three-dimensional imaging device, an augmented reality device, a virtual reality device, a machine vision system, etc., or an electronic device capable of realizing the above functions, etc. The following takes an optoelectronic device as an example to illustrate this embodiment and the following embodiments.

[0057] Based on this, the present invention provides a method for reconstructing depth information. Figure 1 , Figure 1 This is a flowchart diagram of the first embodiment of the depth information reconstruction method of the present application.

[0058] In this embodiment, the depth information reconstruction method includes:

[0059] Step S10: calculating the parallax information of the image to be measured based on a preset structured light array.

[0060] It should be noted that a structured light array refers to a light array with a specific light intensity distribution and phase characteristics generated in a specific way. The image to be measured refers to the image formed by the reflection of the object whose depth information needs to be obtained. For example, the structured light array is projected onto the surface of the object to be measured, and after being reflected by the object, it is received by the receiving end (such as an infrared camera) to form image data. Parallax information represents the position difference of the same object point observed from different angles.

[0061] It should be understood that a structured light array with various characteristic calibration distances can be generated by controlling the light emission position of each light hole in the VCSEL array.

[0062] Step S20: determining initial depth information of the image to be measured according to the disparity information, and reconstructing the disparity information based on the initial depth information to obtain real disparity information.

[0063] It should be noted that the initial depth information is a depth value of each point on the image to be measured (that is, the surface of the object to be measured) obtained by preliminary calculation using a model based on disparity information.

[0064] It is understandable that when the initial depth information is acquired, the depth value of each pixel may be calculated by the triangulation principle in depth detection.

[0065] Step S30 , establishing a depth information joint reconstruction calculation model according to the real disparity information and the characteristic calibration distance of the structured light array.

[0066] It should be noted that the real disparity information is the disparity information reconstructed by the corresponding depth information calculation model and solution method (such as solving the L1 regularization problem) taking into account multiple influencing factors such as system noise, depth measurement error and temperature drift. Compared with the initial disparity information, it can more accurately reflect the actual position difference of the same object point observed from different angles. The characteristic calibration distance is a distance feature with calibration significance presented by the structured light projected in the far field through a specific structured light array generation method (such as using independently controllable VCSEL arrays and metasurface arrays and other components). The depth information joint reconstruction calculation model is a model that integrates key elements such as real disparity information and the characteristic calibration distance of the structured light array. It is built based on mathematical expressions through a series of mathematical relationships and operation rules to accurately calculate and reconstruct depth information.

[0067] It should be understood that the depth information joint reconstruction calculation model can be a linear model based on multiple sets of data, an optimization model including regularization terms, or a hybrid model based on physics and data-driven.

[0068] Step S40: reconstruct the initial depth information through the depth information joint reconstruction calculation model to output target depth information.

[0069] It is important to understand that the target depth information is the final depth information output after a series of operations such as reconstruction and optimization of the initial depth information by the depth information joint reconstruction calculation model. Through the calculation and adjustment of the depth information joint reconstruction calculation model, the error is minimized, and the true depth of each point on the surface of the object in the image to be tested in three-dimensional space can be more accurately reflected, which can provide reliable data support for the subsequent application of structured light-based three-dimensional perception technology in many fields such as artificial intelligence, face recognition, and machine vision.

[0070] In one example, reference Figure 2 , Figure 2This is a module diagram of the optoelectronic device of the present application. The optoelectronic device module includes a feature calibration distance structured light array generation module and a depth information joint reconstruction calculation module. The feature calibration distance structured light array generation module includes an independently controllable VCSEL array and a metasurface array. The independently controllable VCSEL array can realize independent control of the light-emitting position of the light holes of the VCSEL array, and the independently controllable VCSEL array is used as a light source to project onto the metasurface array, so as to project a structured light array with a feature calibration distance in the far field; the depth information joint reconstruction calculation module includes establishing a depth information calculation model based on the L1 regularization problem, establishing a depth information joint reconstruction calculation model based on the feature calibration distance, using a specific L1 regularization optimization algorithm to jointly reconstruct the depth information, and outputting the jointly reconstructed high-precision depth information. The diversity of speckle feature calibration distances can be used to provide additional and more flexible depth reference information for parallax calculations, making full use of more constraints to effectively reduce the cumulative effect of errors and significantly improve the accuracy of depth measurement. The accuracy and reliability of depth measurement are enhanced, and the practical application of high-precision structured light systems in artificial intelligence, face recognition, machine vision and other fields is further expanded, promoting the further development of 3D imaging-based consumer electronics technology.

[0071] In this embodiment, the disparity information of the image to be measured is calculated based on a preset structured light array; the initial depth information of the image to be measured is determined based on the disparity information, and the disparity information is reconstructed based on the initial depth information to obtain real disparity information; a depth information joint reconstruction calculation model is established based on the real disparity information and the characteristic calibration distance of the structured light array; the initial depth information is reconstructed through the depth information joint reconstruction calculation model, and the target depth information is output. By utilizing the diversity of the characteristic calibration distance, more flexible depth reference information is provided for disparity calculation, and more constraints are fully utilized when calculating the depth through the depth information joint reconstruction model, effectively reducing the cumulative effect of errors, and improving the depth measurement accuracy, thereby enhancing the accuracy and reliability of the depth measurement.

[0072] Reference Figure 3 , Figure 3 This is a flow chart of the second embodiment of the depth information reconstruction method of the present application. Based on the above-mentioned first embodiment, the second embodiment of the depth information reconstruction method of the present application is proposed.

[0073] In the second embodiment, the step S10 includes:

[0074] Step S101 , projecting the structured light array onto the image to be measured and a plane at a specific distance respectively, to generate a deformed image and a reference image.

[0075] It should be understood that the image to be measured refers to the image corresponding to the object whose depth information needs to be obtained. When the structured light array is projected onto the surface of the object to be measured, it is reflected and scattered by the surface of the object, and then captured by the corresponding receiving device (such as an infrared camera, etc.) The image contains the appearance information of the object itself, such as texture and shape, as well as the characteristics presented after the interaction between the structured light and the object. The plane at a specific distance is an artificially set plane with a fixed relative position and a known distance to the structured light projection system. Its function is to serve as a standard reference surface. When the structured light array is projected onto the surface of the object to be measured, the shape, texture, and different material characteristics of the object will cause the structured light to reflect, scatter, refract, and other complex optical phenomena, resulting in changes in the original light intensity distribution, phase and other characteristics of the structured light. The changed image captured by the receiving device is called a deformed image.

[0076] It is understandable that in the field of computer vision and 3D reconstruction, parallax is a key concept, which indicates the position difference of the same object point observed from different angles. In the speckle structured light system, parallax is used to calculate the depth information of the object surface. In order to obtain the parallax parameters, the calibrated speckle pattern needs to be projected onto a plane at a specific distance as a reference image; then, the speckle pattern is projected onto the surface of the object to be measured, and the speckle is reflected back to the receiving end by the object to generate a deformed image.

[0077] Furthermore, in order to improve the adaptability and flexibility with the image to be tested, before step S101, it also includes: obtaining measurement information of the image to be tested; adjusting the parameters of the structured light array generation module according to the measurement information, and generating a structured light array with a characteristic calibration distance based on the adjusted structured light array generation module.

[0078] It should be understood that the measurement information is the relevant data and feature description obtained after analyzing and processing the image to be measured, such as the shape and contour of the object in the image, the texture characteristics of the object's surface, the depth and material of the object, etc.

[0079] It can be understood that the structured light queue generation module includes an independently controllable VCSEL array and a metasurface array. The independently controllable VCSEL array can realize independent control of the light emitting position of the light hole of the VCSEL array, and the independently controllable VCSEL array can be used as a light source to project to the metasurface array, which is used to project a coded structured light array with a characteristic calibration distance in the far field. The metasurface structure unit of the metasurface array adopts any one of silicon on insulator material, SiO2-Si material or GaAs material, and its structural form is any one of a cylindrical structure or a prismatic structure.

[0080] It should be noted that when adjusting the parameters of the structured light array generation module, if the measurement information shows that the surface texture of the object is complex and the depth varies greatly, it is necessary to adjust the luminous intensity of some light holes in the VCSEL array so that a clearer and more recognizable structured light pattern can be formed on the complex texture surface; or change the luminous position of the light hole so that the structured light can more evenly cover different depth areas on the surface of the object. At the same time, for the metasurface array, it is also necessary to adjust the structural parameters (such as length, width, etc.) of its nanocolumn unit according to the situation to optimize its phase modulation effect on light, so that the generated structured light array has a more suitable feature calibration distance and better adapts to the depth detection requirements of the object to be measured.

[0081] In one example, reference Figure 4 and Figure 5 , Figure 4 This is a scanning electron microscope image of a supersurface array of the present application. Figure 5 Schematic diagram of a coded structured light array with characteristic calibration distances generated for this application. Figure 4 The nanopillar units of the metasurface array are shown in Figure 1. Phase modulation is performed by changing the length and width of the metasurface nanopillar unit structure to generate structured light with a characteristic calibration distance in the far field. The generated coded structured light array with a characteristic calibration distance is shown in Figure 1. Figure 5 As shown, the replicated and expanded speckle of a single point in the coded structured light array corresponds one-to-one to the light-emitting position of the light hole in the independently controllable VCSEL array.

[0082] Step S102: selecting a speckle in the deformed image as a center to be measured, and selecting a speckle image block based on the center to be measured.

[0083] It should be noted that the speckle image block is an image portion including a selected speckle (center to be measured) and a certain area around it.

[0084] It is understandable that in the deformed image, a scattered spot can be selected from a plurality of scattered spots as the center to be measured based on a variety of factors, such as the brightness, contrast, position in the image (such as being in the center of the region of interest), etc., through a specific algorithm or manual recognition. Scattered spots that are easy to identify and representative in the images before and after deformation can also be selected as the center to be measured.

[0085] Step S103: identifying a matching image block corresponding to the speckle image block from the reference image according to an image matching algorithm, and calculating an offset between the speckle image block and the matching image block to obtain disparity information.

[0086] It can be understood that the process of calculating the disparity involves selecting a speckle as the center, and selecting a corresponding speckle image block based on the center, searching on the reference image through a specific image matching algorithm, and then identifying the corresponding matching image block, and calculating the offset between the two image blocks to determine the disparity parameters of the speckle. The realization of this process relies on precise image processing and matching algorithms to ensure that matching image blocks can be accurately identified and the disparity can be calculated.

[0087] It should be understood that the image matching algorithm may be at least one of a normalized cross-correlation algorithm, a squared difference algorithm, and a convolutional neural network-based matching algorithm.

[0088] In this embodiment, the structured light array is projected onto the image to be measured and the plane at a specific distance respectively to generate a deformed image and a reference image; a speckle is selected as the center to be measured in the deformed image, and a speckle image block is selected based on the center to be measured; a matching image block corresponding to the speckle image block is identified from the reference image according to an image matching algorithm, and the offset between the speckle image block and the matching image block is calculated to obtain disparity information. By generating a deformed image and a reference image to calculate the disparity, the information after the interaction between the structured light and the object can be fully utilized to accurately obtain the position difference information of different points on the surface of the object at different viewing angles.

[0089] Reference Figure 6 , Figure 6 This is a flow chart of the third embodiment of the depth information reconstruction method of the present application. Based on the above second embodiment, the third embodiment of the depth information reconstruction method of the present application is proposed.

[0090] In the third embodiment, the step S20 includes:

[0091] Step S201 : calculating the depth value of each pixel in the image to be tested according to the disparity information to obtain initial depth information.

[0092] In one example, according to the triangulation principle in depth detection, the depth value of each pixel can be calculated using the following formula: :

[0093]

[0094] in, , and Represents the three built-in parameters of baseline, focal length and distance to the reference plane. Indicates parallax.

[0095] Step S202 , obtaining a temperature drift error, and establishing a depth information calculation model through the initial depth information, the parallax information and the temperature drift error.

[0096] It should be noted that before obtaining the temperature drift error, a mathematical model can be established through experiments to describe the relationship between the temperature drift term and temperature, and a temperature sensor can be deployed in the environment where the relevant imaging device or system is located to measure the ambient temperature in real time and calculate the temperature drift error of the current environment.

[0097] It is understandable that when establishing a depth information calculation model using the initial depth information, the parallax information and the temperature drift error, a depth information calculation model that does not consider the temperature drift term can be established first, and then the temperature drift term can be introduced to obtain a depth information calculation model that considers the temperature drift term.

[0098] In one example, the depth information calculation model without considering the temperature drift term is expressed as follows:

[0099]

[0100] in, represents the calculated initial depth information (measured depth information), Indicates the disparity information that needs to be reconstructed, Includes system noise and depth measurement errors. is the system observation matrix, which represents the geometric relationship between depth and disparity, as follows:

[0101]

[0102] in, , and Represents the three built-in parameters of baseline, focal length and distance to the reference plane.

[0103] Reconstructing disparity information This can be achieved by solving the following L1 regularization problem:

[0104]

[0105] in, represents the reconstructed disparity information, represents the Frobenius norm of the matrix, is the regularization parameter.

[0106] The depth information calculation model considering the temperature drift term is expressed as follows:

[0107]

[0108] in, It represents the error introduced by temperature drift, that is, the impact of temperature change on the system.

[0109] Next, a mathematical model needs to be established to describe the temperature drift term With temperature Assume that this relationship can be represented by a simple linear model (a more complex model can be established according to the actual situation):

[0110]

[0111] in, is the temperature sensitivity coefficient, which indicates the depth information deviation caused by unit temperature change; Indicates the reference temperature.

[0112] Combining the above formula, we can further obtain the depth information calculation model considering the temperature drift term:

[0113]

[0114] To measure the value from the depth information with temperature drift Reconstructing the true disparity information , which needs to be optimized.

[0115] Step S203: Solve the depth information calculation model using regularization to obtain real disparity information.

[0116] It is important to understand that when solving complex models such as deep information calculation models that may contain noise and uncertainty, by adding regularization terms to the objective function, the parameters of the model are constrained so that the model will not be too complex while fitting the data, thereby being able to more stably and accurately solve results that meet actual conditions.

[0117] It is understandable that after constructing the objective function containing the L1 regularization term for the depth information calculation model, the objective function can be solved by an iterative algorithm, such as a gradient descent algorithm or a proximal gradient descent algorithm.

[0118] In one example, the depth information is measured with a temperature drift term. Reconstructing the true disparity information , the original L1 regularization problem is optimized to include the temperature drift correction term. The optimized L1 regularization problem is expressed as:

[0119]

[0120] By solving the above optimization problem, we can obtain the reconstructed real parallax information for temperature drift correction. .

[0121] In this embodiment, the depth value of each pixel in the image to be measured is calculated according to the disparity information to obtain the initial depth information; the temperature drift error is obtained, and a depth information calculation model is established through the initial depth information, the disparity information and the temperature drift error; the depth information calculation model is solved by regularization to obtain the real disparity information. By considering the temperature drift error, the depth information calculation model is made more accurate, and the measurement error caused by temperature change is effectively reduced.

[0122] Reference Figure 7 , Figure 7 This is a flow chart of a fourth embodiment of a depth information reconstruction method of the present application. Based on the third embodiment, a fourth embodiment of a depth information reconstruction method of the present application is proposed.

[0123] In the fourth embodiment, the step S30 includes:

[0124] Step S301 : calculating the number of characteristic calibration distances of the structured light array, and determining the position data of scattered spots in the structured light array based on the number.

[0125] It should be understood that the coded structured light array with feature calibration distance generated by the structured light array generation module provides more variable feature calibration distances, which is essentially different from the traditional speckle spacing and scale-free structured light system.

[0126] It can be understood that the number of characteristic calibration distances may indicate that there are several sets of speckle position data for disparity information reconstruction.

[0127] In one example, reference Figure 8 , Fig. 9 , Fig.10 and Fig.11 . Figure 8 Schematic diagram of the structured light array generated for a single VCSEL aperture. Fig. 9 Schematic diagram of a coded structured light array with characteristic calibrated distances generated for four VCSEL apertures. Fig.10 Schematic diagram of the coded structured light array with characteristic calibrated distance generated for a 3×3 VCSEL aperture. Fig.11 Schematic diagram of the characteristic calibration distance calculation for the N×M VCSEL array aperture generation. Figure 8 As a comparison, the structured light array generated by a single VCSEL light hole in Fig. 9 and Fig.10There are three and five calibrated unique distances in , and as the size of the VCSEL array increases, the diversity of the calibrated unique distances also increases. (like , , )and (like , , , , ) represents a variable feature calibration distance. Fig.10 The number of characteristic calibration distances from the structured light to each row is marked under each column. For example, the numbers of characteristic calibration distances from the structured light in the first row and the first column to each structured light are 2, 2, and 1, respectively. Fig.11 Demonstrated by N×M VCSEL array ( ) is a schematic diagram of the feature calibration distance generated by , and its calculation formula is as follows:

[0128]

[0129] in, and represent the number of rows and columns of the VCSEL array respectively. A quantity representing the feature calibration distance.

[0130] Step S302: establishing a depth information joint reconstruction calculation model according to the position data and the real parallax information.

[0131] In one example, considering the different i introduced by the characteristic calibration distance , that is, there are i sets of scattered speckle position data for disparity information reconstruction, and the reconstruction model of i sets of data can be expressed as:

[0132]

[0133] in, represents the measured depth information of the i-th group of data calculated by the image matching algorithm, represents the corresponding real disparity information, represents the observation matrix of this set of data, and They respectively represent the difference between the corresponding measured depth information and the actual depth information, including noise, system error, calibration error, and temperature drift noise.

[0134] The calculation model of joint reconstruction of depth information considering temperature drift is expressed as:

[0135]

[0136] The above formula can be compactly written as:

[0137]

[0138] in,

[0139]

[0140] Jointly reconstruct true depth information , can be reconstructed through an L1 regularized joint optimization problem.

[0141] In the fourth embodiment, the step S40 includes:

[0142] Step S401: determining a joint optimization problem of the initial depth information according to the depth information joint reconstruction calculation model.

[0143] It is understandable that the core of the joint optimization problem of initial depth information based on the joint reconstruction calculation model of depth information is to find an optimal adjustment method so that the initial depth information can be as close to the real depth information as possible while considering various influencing factors (such as parallax information, error factors, etc.). For example, under the model framework, an optimization problem is constructed and solved by defining appropriate objective functions and constraints.

[0144] In one example, jointly reconstructing the true depth information , can be reconstructed by solving the following L1 regularized joint optimization problem:

[0145]

[0146]

[0147] in, represents the jth column.

[0148] Step S402: Solve the joint optimization problem using a regularization algorithm and output target depth information.

[0149] It should be understood that when solving the joint optimization problem, it can be solved by the Iterative Threshold Algorithm (ITA) and the Complex Approximated Message Passing (CAMP) algorithm. The ITA and CAMP algorithms are effective methods for solving L1 regularized optimization problems. They are both iterative algorithms that gradually approach the optimal solution by continuously updating the estimated values ​​of the variables. These algorithms utilize the characteristics of the regularization term and can show good accuracy and efficiency when dealing with large-scale problems and problems with special structures. Specifically, the ITA algorithm iteratively updates the values ​​of the variables and applies threshold operations to enforce sparsity, gradually reducing the value of the objective function, and finally converging to a solution that satisfies the L1 regularization constraint. Minimize the objective function in an iterative manner and gradually update The value of is increased until the convergence condition or the specified number of iterations is reached; the CAMP algorithm is an algorithm based on message passing, which approaches the optimal solution by iteratively passing and updating messages. It is more effective in dealing with problems with complex structures and dependencies.

[0150] It is understandable that when solving the joint optimization problem, other regularization algorithms can also be used to solve the L1 regularization optimization problem, such as gradient descent method, coordinate descent method, etc. In specific implementation, a suitable solution algorithm can be selected according to the characteristics of the specific problem, constraints, computing resources and other factors, and the corresponding parameters can be adjusted to achieve the best performance.

[0151] In one example, taking the CAMP algorithm as an example, the process of solving the regularized joint optimization problem based on feature calibration distance is shown in the following table.

[0152] Table 1 Regularized joint optimization reconstruction algorithm based on CAMP.

[0153]

[0154] in, is the coefficient, is the intermediate parameter, R and I are the real and imaginary parts respectively.

[0155] It should be noted that the above CAMP-based regularized joint optimization reconstruction algorithm is only a preferred embodiment of the present application and is not intended to limit the present application.

[0156] In this embodiment, the diversity of feature calibration distances brings multiple sets of scattered speckle position data. By incorporating these data into the calculation model, the constraints for calculating depth information are increased, effectively reducing the error accumulation effect. And by solving the joint optimization problem through the regularization algorithm, the degree of fit of the model to the data and the complexity of the model can be better balanced to avoid overfitting, thereby improving the accuracy of depth information reconstruction, making the output target depth information more accurate and reliable, and meeting the application needs of structured light-based three-dimensional perception technology in various fields.

[0157] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the depth information reconstruction method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0158] This application also provides a depth information reconstruction device, please refer to Fig.12 , the depth information reconstruction device comprises:

[0159] A parallax calculation module 10, used for calculating the parallax information of the image to be measured based on a preset structured light array;

[0160] A disparity reconstruction module 20, configured to determine initial depth information of the image to be measured according to the disparity information, and reconstruct the disparity information based on the initial depth information to obtain real disparity information;

[0161] A model building module 30, used to establish a depth information joint reconstruction calculation model according to the real disparity information and the characteristic calibration distance of the structured light array;

[0162] The information reconstruction module 40 is used to reconstruct the initial depth information through the depth information joint reconstruction calculation model and output target depth information.

[0163] The depth information reconstruction device provided by the present application adopts the depth information reconstruction method in the above embodiment, which can solve the technical problem that the structured light projected by the traditional structured light system is statically fixed, resulting in errors in parallax calculation. Compared with the prior art, the beneficial effects of the depth information reconstruction device provided by the present application are the same as the beneficial effects of the depth information reconstruction method provided by the above embodiment, and the other technical features in the depth information reconstruction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0164] The present application provides a depth information reconstruction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the depth information reconstruction method in the above-mentioned embodiment one.

[0165] Reference below Fig.13 , which shows a schematic diagram of the structure of a depth information reconstruction device suitable for implementing the embodiment of the present application. The depth information reconstruction device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.13 The depth information reconstruction device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0166] like Fig.13 As shown, the depth information reconstruction device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the depth information reconstruction device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the depth information reconstruction device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a depth information reconstruction device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0167] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0168] The depth information reconstruction device provided by the present application adopts the depth information reconstruction method in the above embodiment, which can solve the technical problem that the structured light projected by the traditional structured light system is statically fixed, resulting in errors in parallax calculation. Compared with the prior art, the beneficial effects of the depth information reconstruction device provided by the present application are the same as the beneficial effects of the depth information reconstruction method provided by the above embodiment, and the other technical features in the depth information reconstruction device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0169] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0170] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0171] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the depth information reconstruction method in the above-mentioned embodiment.

[0172] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0173] The computer-readable storage medium may be included in the depth information reconstruction device; or may exist independently without being assembled into the depth information reconstruction device.

[0174] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the depth information reconstruction device, the depth information reconstruction device executes the depth information reconstruction method described above.

[0175] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0176] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0177] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0178] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned depth information reconstruction method, and can solve the technical problem that the structured light projected by the traditional structured light system is statically fixed, resulting in errors in parallax calculation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the depth information reconstruction method provided by the above-mentioned embodiment, and will not be repeated here.

[0179] The present application also provides a computer program product, including a computer program, which implements the steps of the depth information reconstruction method as described above when executed by a processor.

[0180] The computer program product provided by the present application can solve the technical problem that the structured light projected by the traditional structured light system is statically fixed, resulting in errors in parallax calculation. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the depth information reconstruction method provided by the above embodiment, and will not be repeated here.

[0181] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A depth information reconstruction method, characterized in that: The depth information reconstruction method comprises: Calculating the parallax information of the image to be measured based on a preset structured light array; Determining initial depth information of the image to be tested according to the disparity information, and reconstructing the disparity information based on the initial depth information to obtain real disparity information; A depth information joint reconstruction calculation model is established according to the real disparity information and the characteristic calibration distance of the structured light array, wherein the characteristic calibration distance is a distance feature with calibration significance presented in the structured light projected in the far field through a specific structured light array generation method; Reconstructing the initial depth information through the depth information joint reconstruction calculation model, and outputting target depth information; The step of determining the initial depth information of the image to be tested according to the disparity information, and reconstructing the disparity information based on the initial depth information to obtain real disparity information includes: Calculating the depth value of each pixel in the image to be tested according to the disparity information to obtain initial depth information; Acquire a temperature drift error, and establish a depth information calculation model through the initial depth information, the parallax information, and the temperature drift error; Solving the depth information calculation model by using regularization to obtain real disparity information; The step of establishing a depth information joint reconstruction calculation model according to the real disparity information and the characteristic calibration distance of the structured light array comprises: Calculating the number of characteristic calibration distances of the structured light array, and determining the position data of scattered spots in the structured light array based on the number, wherein the number of characteristic calibration distances indicates the number of groups of position data of scattered spots for parallax information reconstruction; A depth information joint reconstruction calculation model is established according to the position data and the real disparity information.

2. The depth information reconstruction method according to claim 1, characterized in that: The step of calculating the parallax information of the image to be measured based on the preset structured light array includes: The structured light array is projected onto the image to be measured and a plane at a specific distance, respectively, to generate a deformed image and a reference image, wherein the plane at the specific distance is an artificially set plane with a fixed relative position to the structured light projection system and a known distance; Selecting a speckle in the deformed image as a center to be measured, and selecting a speckle image block based on the center to be measured; A matching image block corresponding to the speckle image block is identified from the reference image according to an image matching algorithm, and an offset between the speckle image block and the matching image block is calculated to obtain disparity information.

3. The depth information reconstruction method according to claim 2, characterized in that: Before the step of projecting the structured light array onto the image to be measured and the plane at a specific distance to generate the deformed image and the reference image, the method further includes: Acquiring measurement information of the image to be measured; The parameters of the structured light array generation module are adjusted according to the measurement information, and a structured light array with a characteristic calibration distance is generated based on the adjusted structured light array generation module.

4. The depth information reconstruction method according to claim 1, characterized in that: The step of reconstructing the initial depth information through the depth information joint reconstruction calculation model and outputting target depth information includes: Determining a joint optimization problem of the initial depth information according to the depth information joint reconstruction calculation model; The joint optimization problem is solved using a regularization algorithm, and target depth information is output.

5. A depth information reconstruction device, characterized in that: The device comprises: A disparity calculation module, used for calculating the disparity information of the image to be measured based on a preset structured light array; A disparity reconstruction module, used to determine the initial depth information of the image to be measured according to the disparity information, and reconstruct the disparity information based on the initial depth information to obtain real disparity information; A model building module, used to establish a depth information joint reconstruction calculation model according to the real disparity information and the characteristic calibration distance of the structured light array, wherein the characteristic calibration distance is a distance feature with calibration significance presented in the structured light projected in the far field through a specific structured light array generation method; An information reconstruction module, used to reconstruct the initial depth information through the depth information joint reconstruction calculation model, and output target depth information; The disparity reconstruction module is further used to calculate the depth value of each pixel in the image to be measured according to the disparity information to obtain initial depth information; obtain temperature drift error, and establish a depth information calculation model through the initial depth information, the disparity information and the temperature drift error; and solve the depth information calculation model by regularization to obtain real disparity information; The model building module is also used to calculate the number of characteristic calibration distances of the structured light array, and determine the position data of scattered spots in the structured light array based on the number, wherein the number of characteristic calibration distances indicates that there are several groups of position data of scattered spots for disparity information reconstruction; and establish a depth information joint reconstruction calculation model based on the position data and the real disparity information.

6. A depth information reconstruction device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the depth information reconstruction method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the depth information reconstruction method according to any one of claims 1 to 4 are implemented.

8. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the depth information reconstruction method according to any one of claims 1 to 4 are implemented.

Images