Depth reconstruction method, device and storage medium

Abstract

The present application discloses a method, device and computer-readable storage medium for depth reconstruction. The method includes: extracting features of the current frame to obtain a feature matrix of the current frame, where the current frame is an image obtained by spatial projection of a preset structured light pattern being processed; using the feature matrix of the current frame and the feature matrix of the previous frame for prediction to obtain a predicted disparity map of the current frame; using the feature matrix of the current frame and the feature matrix of the structured light pattern to correct the predicted disparity map of the current frame to obtain a corrected disparity map of the current frame. In the above manner, the amount of calculation required for depth reconstruction can be reduced, thereby improving the frame rate of reconstruction.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a depth reconstruction method, device and computer-readable storage medium. Background Art

[0002] Restoring the depth information of three-dimensional dynamic scenes is an important issue in the field of computer vision. The three-dimensional information of dynamic scenes is in high demand in application scenarios such as AR / VR, motion analysis, and digital medicine. Many further analyses and recognitions require real-time acquisition of high-precision, dense three-dimensional information as the basis of the algorithm. Structured light is a way to implement monocular stereo vision. A structured light system generally includes a light source (also called a projection light source) and a single camera. The light source projects a designed structured light pattern, or a structured light pattern. If there is an object in space, the structured light is projected on the surface of the object, and a single camera collects the projection image of the structured light. The projection of the structured light will be distorted with the depth of each part of the surface of the object (that is, the distance from the structured light system). Analyzing the distortion can obtain the position and depth information of the object.

[0003] Although the depth of the image is more accurate and robust after adding projection, it is still very difficult to achieve real-time online requirements for dynamic scenes. In each frame of the image, the deformation pattern needs to be globally matched and optimized with the original pattern. This amount of calculation will cause the reconstruction frame rate to drop, making it difficult to meet real-time requirements. Summary of the invention

[0004] The present application provides a depth reconstruction method, device and computer-readable storage medium, which can solve the problem that the existing structured light depth reconstruction is difficult to dynamically reconstruct parallax images in real time.

[0005] In order to solve the above technical problems, a technical solution adopted by the present application is to provide a depth reconstruction method. The method includes: extracting the features of the current frame to obtain the feature matrix of the current frame, where the current frame is the image obtained by spatial projection of the preset structured light pattern being processed; using the feature matrix of the current frame and the feature matrix of the previous frame to predict, to obtain a predicted disparity map of the current frame; using the feature matrix of the current frame and the feature matrix of the structured light pattern to correct the predicted disparity map of the current frame, to obtain a corrected disparity map of the current frame.

[0006] Optionally, prediction is performed using a feature matrix of the current frame and a feature matrix of the previous frame to obtain a predicted disparity map of the current frame, including: calculating a pattern flow between the current frame and the previous frame using the feature matrix of the current frame and the previous frame, the pattern flow being an optical flow caused by changes in the spatial projection of the structured light pattern between the current frame and the previous frame; and transforming a corrected disparity map of the previous frame using the pattern flow to obtain a predicted disparity map of the current frame.

[0007] Optionally, using the feature matrix of the current frame and the feature matrix of the previous frame to calculate the pattern flow between the current frame and the previous frame includes: obtaining the pattern flow by iterative calculation through a first loop update block, the input of the first loop update block includes the feature matrix of the current frame and the feature matrix of the previous frame, and using the output result of the first loop update block this time as the input of the next iterative calculation until the number of iterations reaches a preset value.

[0008] Optionally, a pattern stream is used to transform a corrected disparity map of a previous frame to obtain a predicted disparity map of a current frame, including: determining a first difference between a disparity image of a current frame and a disparity image of a previous frame based on the pattern stream, the disparity image of the current frame being a disparity image between the current frame and a structured light pattern, and the disparity image of the previous frame being a disparity image between the previous frame and the structured light pattern; and summing the first difference and the corrected disparity map of the previous frame to obtain a predicted disparity map of the current frame.

[0009] Optionally, using the feature matrix of the current frame and the feature matrix of the structured light pattern to correct the predicted disparity map of the current frame includes: using the feature matrix of the current frame and the feature matrix of the structured light pattern to perform prediction to obtain a first disparity map of the current frame; using the first disparity map of the current frame and the predicted disparity map of the current frame to obtain a second disparity map of the current frame; and using the feature matrix of the current frame and the feature matrix of the structured light pattern to correct the second disparity map of the current frame.

[0010] Optionally, obtaining the second disparity map of the current frame by using the first disparity map of the current frame and the predicted disparity map of the current frame includes: fusing the first disparity map of the current frame with the predicted disparity map of the current frame to obtain the second disparity map of the current frame.

[0011] Optionally, using the feature matrix of the current frame and the feature matrix of the structured light pattern to correct the second disparity map of the current frame includes: obtaining the corrected disparity map of the current frame by iterative calculation through a second loop update block, the input of the second loop update block includes the second disparity map of the current frame, the feature matrix of the current frame and the feature matrix of the structured light pattern, and using the output result of the second loop update block this time as the input of the next iterative calculation until the number of iterations reaches a preset value.

[0012] Optionally, the structured light pattern is a pseudo-random binary pattern and is unique in one of the row and column directions and is periodically repeated in the other direction.

[0013] Optionally, it further includes: using the photometric error between the current frame and the structured light pattern to perform self-supervised training on at least one of the feature extraction module, the first prediction module and the correction module, the feature extraction module is used to extract features of the current frame, the first prediction module is used to use the feature matrix of the current frame and the feature matrix of the previous frame for prediction, and the correction module is used to use the feature matrix of the current frame and the feature matrix of the structured light pattern to correct the predicted disparity map of the current frame.

[0014] Optionally, the method further includes: performing self-supervised training using the corrected disparity map of the current frame output by the correction module.

[0015] Optionally, the loss function of self-supervised training includes a pattern flow loss function, a prediction loss function and a disparity loss function; the loss function used for training the feature extraction module is calculated using the pattern flow loss function, the prediction loss function and the disparity loss function; the loss function used for training the first prediction module includes the pattern flow loss function and the prediction loss function; and the loss function used for training the correction module includes the disparity loss function.

[0016] To solve the above technical problems, another technical solution adopted in the present application is: providing a depth reconstruction device, which includes a feature extraction module, used to extract the features of the current frame and obtain the feature matrix of the current frame, the current frame is the image obtained by spatial projection of the preset structured light pattern currently being processed; a first prediction module, used to use the feature matrix of the current frame and the feature matrix of the previous frame for prediction, to obtain a predicted disparity map of the current frame; a correction module, used to use the feature matrix of the current frame and the feature matrix of the structured light pattern to correct the predicted disparity map of the current frame, to obtain a corrected disparity map of the current frame.

[0017] To solve the above technical problems, another technical solution adopted in the present application is: to provide a depth reconstruction device, which includes a processor and a memory coupled to the processor, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.

[0018] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium storing program instructions, which can implement the above method when executed.

[0019] Through the above-mentioned method, the present application extracts the features of the current frame through the feature extraction module to obtain the feature matrix of the current frame, where the current frame is the image obtained by spatial projection shooting of the preset structured light pattern currently being processed; the feature matrix of the current frame and the feature matrix of the previous frame are used for prediction through the first prediction module to obtain the predicted disparity map of the current frame; the predicted disparity map of the current frame is corrected by the correction module using the feature matrix of the current frame and the feature matrix of the structured light pattern to obtain the corrected disparity map of the current frame. The feature matrix of the previous frame is introduced in the process of calculating the disparity map of the current frame. The timing information brought by the previous frame can effectively reduce the amount of calculation required for depth reconstruction, thereby improving the frame rate of reconstruction and better meeting the real-time requirements of depth reconstruction of dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of an embodiment of the depth reconstruction method of the present application;

[0021] Figure 2 yes Figure 1 The specific process diagram of S2 in FIG.

[0022] Figure 3 It is a schematic diagram of the geometric relationship of the mode flow transformation in an embodiment of the depth reconstruction method of the present application;

[0023] Figure 4 yes Figure 2 The specific process diagram of S22;

[0024] Figure 5 yes Figure 1 The specific process diagram of S3 in the figure;

[0025] Figure 6 This is a schematic diagram of the overall framework of a deep learning network TIDE using an embodiment of the deep reconstruction method of the present application;

[0026] Figure 7 This is a schematic diagram of TIDE's network structure;

[0027] Figure 8 is a schematic structural diagram of an embodiment of a depth reconstruction device of the present application;

[0028] Fig. 9 is a schematic structural diagram of another embodiment of the depth reconstruction device of the present application;

[0029] Fig.10 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] The terms "first", "second", and "third" in this application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first", "second", and "third" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0032] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments without conflict.

[0033] Figure 1 FIG. 1 is a flow chart of an embodiment of the depth reconstruction method of the present application. It should be noted that if there are substantially the same results, this embodiment does not necessarily use Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment may include:

[0034] S1: Extract the features of the current frame and obtain the feature matrix of the current frame.

[0035] This embodiment estimates depth information of continuous frames in a single-camera structured light system. The current frame is an image currently processed and obtained by capturing a spatial projection of a preset structured light pattern.

[0036] In a single-camera structured light system, the projected light source can be regarded as a second camera, and its imaging plane (e.g., a diffractive optical element) is used as a reference pattern, which can form a binocular disparity similar to stereo vision. It is assumed that both the camera and the projected light source are calibrated and corrected. For consecutive frames in a single-camera structured light system, the system can take the camera image as input and provide a disparity map for each frame.

[0037] In this system, the preset structured light mode is P, and the current frame is I t is the camera image acquired at time t, D tis the output disparity image, that is, the structured light pattern P and I t disparity image, where t can be represented by a specific timestamp or a frame number in a continuous frame sequence.

[0038] The structured light pattern can be a designed two-dimensional pattern, or a grating coding pattern.

[0039] The structured light pattern can be a pseudo-random binary pattern, and is unique in one of the row and column directions and periodically repeated in the other direction. For example, the spots in the structured light are unique in the row, that is, the spots in each row are different, and are periodically repeated in the vertical direction, that is, the column direction.

[0040] Specifically, the local contrast normalization (LCN) method can be used to convert I t After being connected with the standardized image, it is input into the feature extraction module. The feature extraction module may include two encoders with the same structure, one for extracting image features and the other for extracting semantic features. The outputs of the two are combined to obtain the feature matrix of the current frame.

[0041] The same feature extraction module can be used to extract the feature matrix of the structured light pattern.

[0042] S2: Use the feature matrix of the current frame and the feature matrix of the previous frame to perform prediction to obtain a predicted disparity map of the current frame.

[0043] In dynamic scenes, as the structured light system and / or objects in the projection space move, camera images acquired at different times will be deformed due to changes in the position and / or depth of the objects. By calculating these deformations and combining the corrected disparity map of the previous frame, the disparity map of the current frame can be predicted. Compared with existing single-frame calculations, such inter-frame estimation can reduce the amount of calculation.

[0044] Optional, such as Figure 2 As shown, S2 specifically includes:

[0045] S21: Calculate the pattern flow between the current frame and the previous frame using the feature matrix of the current frame and the feature matrix of the previous frame.

[0046] Mode Flow F t is the spatial projection of the structured light pattern in the current frame I t With the previous frame I t-1 The optical flow caused by the change between is used to represent the movement of the projected light. The pattern flow can be calculated in a similar way to the optical flow, but the pattern flow is different from the traditional optical flow in the following ways.

[0047] First, optical flow is the movement of physical points, while pattern flow is the displacement of projected rays, which means it will only move along one-dimensional lines, the epipolar lines, and not a 2D image.

[0048] Second, pattern flow under structured light is always easier to compute, because the patterns always have enough features for local matching. Therefore, we can significantly reduce the network size and number of iterations for pattern flow computation.

[0049] Finally, although pattern flow has the aforementioned advantages over optical flow, it does not guarantee that the tracked points in two frames belong to the same physical point. Pattern flow lacks physical correspondence information and cannot be used for motion estimation. However, it can be used to transform from the disparity of the previous frame to the disparity of the current frame.

[0050] Specifically, the pattern stream can be obtained by iterative calculation of the first loop update block, the input of the first loop update block can include the feature matrix of the current frame and the feature matrix of the previous frame, and the output result of the first loop update block this time is used as the input of the next iterative calculation until the number of iterations reaches a preset value. The first loop update block includes a temporal convolution gated recurrent unit (ConvGRU), which is a gated recurrent unit that replaces the fully connected with a convolution. S22: Use the pattern stream to transform the corrected disparity map of the previous frame to obtain a predicted disparity map of the current frame.

[0051] The geometric relationship of mode flow transformation is described below.

[0052] like Figure 3 As shown, consider the point P(x p ,y), the point projected by this light at time t in space is p(X,Y,Z). At time t+1, as the object and / or structured light system in space moves, the point projected by this light in space moves to p+Δp, where Δp=(ΔX,0,ΔZ). Please note that p and p+Δp are not guaranteed to be at the same point on the same object, but on the same projected light. Therefore, p will only move on the epipolar plane, and for a calibrated system, the vertical direction does not need to be considered. f is the focal length of the camera, b is the baseline between the camera and the projected light source, (x,y) is the coordinate on the camera imaging plane, corresponding to the coordinate in the camera image, we get:

[0053]

[0054] Differentiating formula (1), we can get:

[0055]

[0056] Since the projection point only moves along the projection light, it can be known that The mode flow can be represented as a horizontal vector, F t (x,y)=(u,0), where Formula (2) can be further simplified as:

[0057]

[0058] On the other hand, let d be D t (x,y) and D t-1 The difference in pixel values ​​at (x, y) can be obtained based on epipolar geometry: Differentiating d, we get:

[0059]

[0060] According to formula (3) and formula (4), we can know that:

[0061]

[0062] This means that the mode flow is D t With D t-1 Based on this relationship, the modified disparity map of the previous frame can be transformed into the predicted disparity map of the current frame according to the mode flow.

[0063] like Figure 4 As shown, S22 may specifically include:

[0064] S221: Determine a first difference between a disparity image of a current frame and a disparity image of a previous frame based on the mode stream.

[0065] The disparity image D of the current frame t The current frame I t The disparity image between the structured light pattern P and the previous frame’s disparity image D t-1 The previous frame I t-1 and the disparity image between the structured light pattern P. Referring to the above geometric relationship, the first difference value can be calculated according to the pattern flow.

[0066] S222: Summing the first difference value and the corrected disparity map of the previous frame to obtain a predicted disparity map of the current frame.

[0067] The mode flow can explicitly provide the disparity propagation between frames, thereby reducing the correspondence error caused by object motion and improving the robustness of the depth scene to the motion scene.

[0068] S3: Correcting the predicted disparity map of the current frame using the feature matrix of the current frame and the feature matrix of the structured light pattern to obtain a corrected disparity map of the current frame.

[0069] After obtaining the corrected disparity map of the current frame, the depth map of the current frame can be reconstructed by combining the parameters of the structured light system and using a common depth reconstruction algorithm.

[0070] like Figure 5 As shown, S3 may specifically include:

[0071] S31: using the feature matrix of the current frame and the feature matrix of the structured light pattern to perform prediction, and obtain a first disparity map of the current frame.

[0072] S32: Obtain a second disparity map of the current frame using the first disparity map of the current frame and the predicted disparity map of the current frame.

[0073] In this embodiment, the acquisition of the depth information of the current frame requires the depth information of the previous frame. However, for the first frame, there is no previous frame, so the first disparity map needs to be introduced. The first disparity map is obtained based on the current frame and the structured light pattern prediction, without using the information of the previous frame. For example, semi-global block matching (SGBM) can be used to predict the first disparity map.

[0074] The first disparity map of the current frame may be fused with the predicted disparity map of the current frame to obtain the second disparity map of the current frame.

[0075] To merge the two disparity maps, a small network is used to estimate a soft mask. The input of this network consists of the current frame I t , and Output soft mask M t The value range of is [0,1]. is the predicted disparity map at time t Back-project the projection mode P into the camera space, and the resulting drawn image; is the first disparity map at time t The image drawn by back-projecting the projection mode P to the camera space. The second disparity map at time t is obtained using the following formula:

[0076]

[0077] in, is the second disparity map at time t, is the predicted disparity map at time t, is the first disparity map at time t.

[0078] Alternatively, one may be selected from the first disparity map and the predicted disparity map as the second disparity map of the current frame according to time t. For example, if the frame corresponding to time t is the first frame, the first disparity map is selected as the second disparity map, otherwise the predicted disparity map is selected as the second disparity map.

[0079] S33: Correcting the second disparity map of the current frame using the feature matrix of the current frame and the feature matrix of the structured light pattern.

[0080] Specifically, the corrected disparity map of the current frame is obtained by iterative calculation of the second loop update block, the input of the second loop update block includes the second disparity map of the current frame, the feature matrix of the current frame and the feature matrix of the structured light pattern, and the output result of the second loop update block is used as the input of the next iterative calculation until the number of iterations reaches a preset value. The structure of the second loop update block is the same as that of the first loop update block.

[0081] S4: Performing self-supervised training on at least one of the feature extraction module, the first prediction module, and the correction module using the photometric error between the current frame and the structured light pattern.

[0082] The feature extraction module is used to extract features of the current frame, the first prediction module is used to use the feature matrix of the current frame and the feature matrix of the previous frame for prediction, and the correction module is used to correct the predicted disparity map of the current frame using the feature matrix of the current frame and the feature matrix of the structured light pattern.

[0083] If the training has been completed, this step can be omitted.

[0084] The corrected disparity map of the current frame output by the correction module can be further used for self-supervised training.

[0085] The loss function of supervised training includes the mode flow loss function L F , prediction loss function L C And the disparity loss function L D The calculation methods are described below.

[0086] Mode flow loss function L F :For from I t ,I t-1 The estimated mode flow F t , the associated pixels should theoretically correspond to the same light in the reference pattern (i.e., the structured light pattern), i.e., have the same intensity value. Therefore, the pattern flow F t The luminosity loss is defined as:

[0087]

[0088] Where π is the warping function used to give the relationship between frames, and ρ represents the speckle-based photometric loss between two images, which is defined as follows:

[0089] ρ(I i ,I j )=∑ x,y ∑ u,v∈(x,y) |I i (u,v)-I j (u,v)| C (8)

[0090] Among them, |·| C Represents a smooth Census transform.

[0091] Prediction loss function L C :The goal of the prediction module is to give the correction module as accurate prediction as possible. Therefore, the final corrected disparity map should guide the prediction module in reverse. The prediction loss is the L1 distance between the predicted disparity map and the corrected disparity map:

[0092]

[0093] in is the corrected disparity map at time t.

[0094] Disparity loss function L D : The photometric loss of disparity is similar to pattern flow, except that the reference pattern is warped to the current frame instead of the previous frame image. The fused second disparity map and the final corrected disparity map are used to calculate the disparity loss:

[0095]

[0096] Where λ is a coefficient, and its value can be set to 0.8 according to experiments.

[0097] The loss function used to train the feature extraction module (also called the overall loss function) is calculated using the pattern flow loss function, the prediction loss function, and the disparity loss function. Define (t-T+1:t) as a time window with a size of T. t-T+1:t ,P is the input, For output, the formula for calculating the overall loss function is as follows:

[0098]

[0099] where ω E is a relative weight factor, and its value can be set to 0.2 based on experiments.

[0100] The loss function used to train the first prediction module includes the mode flow loss function L FAnd the prediction loss function L C The loss function used in training the correction module includes the disparity loss function L D .

[0101] Through the implementation of this embodiment, the feature matrix of the previous frame is introduced in the process of calculating the disparity map of the current frame. The timing information brought by the previous frame can effectively reduce the amount of calculation required for depth reconstruction, thereby improving the reconstructed frame rate and better meeting the real-time requirements of dynamic scene depth reconstruction. At the same time, the network using the depth reconstruction method provided by this embodiment does not require additional data truth value training, but self-supervised training, which effectively reduces the difficulty of training.

[0102] A deep learning network using this embodiment can be called Temporally Iterative Disparity Estimation (TIDE), and its overall framework is as follows: Figure 6 The specific network structure is as shown in Figure 7 shown. Figure 6 The Encoder in the figure represents the feature extraction module, the Predictor represents the first prediction module, and the Refiner represents the correction module.

[0103] Figure 7 The upper left part is the feature extraction module, which includes two encoders with the same structure, one for extracting image features and the other for extracting semantic features. t After connecting with the standardized image, it is input into the feature extraction module to obtain the corresponding image features With semantic feature C t .

[0104] Then the features of the current frame and the features of the previous frame are input into the first loop update block to obtain the mode flow of the current frame. Specifically, the image features of the current frame and the image features of the previous frame can be used to calculate the correlation features between the current frame and the previous frame, and then the current mode flow and the semantic feature C are superimposed. t , input into the time convolution gate loop unit. The output of the time convolution gate loop unit is convolved and up-sampled to obtain the updated pattern stream. The updated pattern stream is input into the time convolution gate loop unit as the current pattern stream in the next iterative calculation. This cycle repeats until the number of iterative calculations reaches the preset value. The updated pattern stream obtained by the last iterative calculation is the pattern stream of the current frame. The initial value of the current pattern stream can be set to 0 or other values, such as the pattern stream of the previous frame.

[0105] Based on the mode stream of the current frame, the first difference between the disparity image of the current frame and the disparity image of the previous frame can be determined, and then the first difference and the corrected disparity map of the previous frame are summed to obtain the predicted disparity map of the current frame.

[0106] Figure 7 The upper right part is a small network that estimates the soft mask.

[0107] Using soft mask, The first disparity map predicted by SGBM Fusion, get the second disparity map

[0108] Then The features of the current frame and the features of the structured light pattern are input into the second loop update block to obtain the corrected disparity map of the current frame. Specifically, the image features of the current frame and the image features of the structured light pattern (obtained by inputting the structured light pattern into the feature extraction module) can be used to calculate the associated features between the current frame and the structured light pattern, and then the current disparity map and the semantic feature C are superimposed. t , input into the temporal convolution gate recurrent unit. The output of the temporal convolution gate recurrent unit is convolved and upsampled to obtain the disparity map for this update. The disparity map for this update is input into the temporal convolution gate recurrent unit as the current disparity map in the next iterative calculation. This cycle repeats until the number of iterative calculations reaches the preset value. The updated disparity map obtained by the last iterative calculation is the corrected disparity map of the current frame. The initial value of the current disparity map is

[0109] Figure 8 FIG. 1 is a schematic diagram of the structure of an embodiment of a depth reconstruction device of the present application. Figure 8 As shown, the depth reconstruction device includes a feature extraction module 11, a first prediction module 12 and a correction module 13.

[0110] The feature extraction module 11 is used to extract features of the current frame to obtain a feature matrix of the current frame. The current frame is an image obtained by spatial projection of a preset structured light pattern currently being processed.

[0111] The first prediction module 12 is used to perform prediction using the feature matrix of the current frame and the feature matrix of the previous frame to obtain a predicted disparity map of the current frame.

[0112] The correction module 13 is used to correct the predicted disparity map of the current frame by using the feature matrix of the current frame and the feature matrix of the structured light pattern to obtain a corrected disparity map of the current frame.

[0113] For the functions and possible extensions of each module in this embodiment, reference may be made to the relevant contents of the depth reconstruction method embodiment of the present application.

[0114] Fig. 9 FIG. 1 is a schematic diagram of the structure of another embodiment of the depth reconstruction device of the present application. Fig. 9As shown, the depth reconstruction device includes a processor 21 and a memory 22 coupled to the processor 21 .

[0115] The memory 22 stores program instructions for implementing the method of any of the above embodiments; the processor 21 is used to execute the program instructions stored in the memory 22 to implement the steps of the above method embodiments. The processor 21 may also be referred to as a CPU (Central Processing Unit). The processor 21 may be an integrated circuit chip with signal processing capabilities. The processor 21 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0116] Fig.10 Schematic diagram of the structure of an embodiment of a computer-readable storage medium of the present application. Fig.10 As shown, the computer-readable storage medium 30 of the embodiment of the present application stores program instructions 31, and the program instructions 31 are executed to implement the method provided in the above embodiment of the present application. Among them, the program instructions 31 can form a program file and be stored in the above computer-readable storage medium 30 in the form of a software product, so that a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) executes all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 30 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0117] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0118] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A depth reconstruction method, characterized in that: include: Extracting features of a current frame to obtain a feature matrix of the current frame, wherein the current frame is an image obtained by spatial projection of a preset structured light pattern being processed; Using the feature matrix of the current frame and the feature matrix of the previous frame to perform prediction, to obtain a predicted disparity map of the current frame; Correcting the predicted disparity map of the current frame by using the feature matrix of the current frame and the feature matrix of the structured light pattern to obtain a corrected disparity map of the current frame; The modifying the predicted disparity map of the current frame by using the feature matrix of the current frame and the feature matrix of the structured light pattern includes: Using the feature matrix of the current frame and the feature matrix of the structured light pattern to perform prediction, to obtain a first disparity map of the current frame; Obtaining a second disparity map of the current frame by using the first disparity map of the current frame and the predicted disparity map of the current frame; The second disparity map of the current frame is corrected using the feature matrix of the current frame and the feature matrix of the structured light pattern.

2. The method according to claim 1, characterized in that The step of using the feature matrix of the current frame and the feature matrix of the previous frame to perform prediction to obtain the predicted disparity map of the current frame includes: Calculating a pattern flow between the current frame and the previous frame using a feature matrix of the current frame and a feature matrix of a previous frame, wherein the pattern flow is an optical flow caused by a change in a spatial projection of the structured light pattern between the current frame and the previous frame; The modified disparity map of the previous frame is transformed using the mode stream to obtain the predicted disparity map of the current frame.

3. The method according to claim 2, characterized in that The calculating the mode flow between the current frame and the previous frame by using the feature matrix of the current frame and the feature matrix of the previous frame comprises: The pattern flow is obtained by iterative calculation through a first loop update block, the input of the first loop update block includes the feature matrix of the current frame and the feature matrix of the previous frame, and the output result of the first loop update block this time is used as the input of the next iterative calculation until the number of iterations reaches a preset value.

4. The method according to claim 2, characterized in that: The adopting the mode flow to transform the corrected disparity map of the previous frame to obtain the predicted disparity map of the current frame comprises: Determining a first difference between a disparity image of the current frame and a disparity image of the previous frame based on the pattern stream, wherein the disparity image of the current frame is a disparity image between the current frame and the structured light pattern, and the disparity image of the previous frame is a disparity image between the previous frame and the structured light pattern; The first difference value and the corrected disparity map of the previous frame are summed to obtain a predicted disparity map of the current frame.

5. The method according to claim 1, characterized in that The obtaining the second disparity map of the current frame by using the first disparity map of the current frame and the predicted disparity map of the current frame comprises: The first disparity map of the current frame is fused with the predicted disparity map of the current frame to obtain a second disparity map of the current frame.

6. The method according to claim 1, characterized in that The correcting the second disparity map of the current frame by using the feature matrix of the current frame and the feature matrix of the structured light pattern comprises: The corrected disparity map of the current frame is obtained by iterative calculation through the second loop update block, the input of the second loop update block includes the second disparity map of the current frame, the feature matrix of the current frame and the feature matrix of the structured light pattern, and the output result of the second loop update block this time is used as the input of the next iterative calculation until the number of iterations reaches a preset value.

7. The method according to any one of claims 1 to 6, characterized in that The structured light pattern is a pseudo-random binary pattern and is unique in one of the row and column directions and is periodically repeated in the other direction.

8. The method according to any one of claims 1 to 6, characterized in that Further including: The photometric error between the current frame and the structured light pattern is used to perform self-supervised training on at least one of a feature extraction module, a first prediction module and a correction module, wherein the feature extraction module is used to extract features of the current frame, the first prediction module is used to perform prediction using a feature matrix of the current frame and a feature matrix of a previous frame, and the correction module is used to correct the predicted disparity map of the current frame using a feature matrix of the current frame and a feature matrix of the structured light pattern.

9. The method according to claim 8, characterized in that Further including: The corrected disparity map of the current frame output by the correction module is used for self-supervised training.

10. The method according to claim 9, characterized in that The loss function of the self-supervised training includes a pattern flow loss function, a prediction loss function and a disparity loss function. The loss function used to train the feature extraction module is calculated using the pattern flow loss function, the prediction loss function and the disparity loss function. The loss function used to train the first prediction module includes the pattern flow loss function and the prediction loss function. The loss function used to train the correction module includes the disparity loss function.

11. A depth reconstruction device, characterized in that: include: A feature extraction module, used to extract features of a current frame to obtain a feature matrix of the current frame, wherein the current frame is an image obtained by spatial projection of a preset structured light pattern being processed; A first prediction module, configured to use a feature matrix of the current frame and a feature matrix of a previous frame to perform prediction to obtain a predicted disparity map of the current frame; A correction module, configured to correct the predicted disparity map of the current frame by using the feature matrix of the current frame and the feature matrix of the structured light pattern to obtain a corrected disparity map of the current frame; The correction module uses the feature matrix of the current frame and the feature matrix of the structured light pattern to correct the predicted disparity map of the current frame, and the method of obtaining the corrected disparity map of the current frame includes: using the feature matrix of the current frame and the feature matrix of the structured light pattern to perform prediction to obtain a first disparity map of the current frame; A second disparity map of the current frame is obtained by using the first disparity map of the current frame and the predicted disparity map of the current frame; and the second disparity map of the current frame is corrected by using a feature matrix of the current frame and a feature matrix of the structured light pattern.

12. A depth reconstruction device, characterized in that: The invention comprises a processor and a memory coupled to the processor, wherein: The memory stores program instructions; The processor is configured to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The storage medium stores program instructions, and when the program instructions are executed, the method according to any one of claims 1 to 10 is implemented.

Images